{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3edcb8e4",
   "metadata": {},
   "source": [
    "# Sparse Operation Kit #\n",
    "---\n",
    "This notebook introduces what is sparse operation kit and how to use it to accerlerate the recommander system's training process.\n",
    "\n",
    "Sparse Opertion Kit (hereafter SOK) is a toolkit aiming at wrapping those effecient algorithms / implementations used in recommendation scenarios, which includes many sparse operations, into a user-friendly library. When user wants to leverage those GPU-accerlerated algorithms to speed up their application, they can quickly start from this Python toolkit. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9fc6d79",
   "metadata": {},
   "source": [
    "## Documents ##\n",
    "Documentation: https://nvidia-merlin.github.io/HugeCTR/sparse_operation_kit/master/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bd97608",
   "metadata": {},
   "source": [
    "## Menu ##\n",
    "1. **Installation**\n",
    "2. **Single-node, Multi-GPUs synchronized training**\n",
    "3. **Multi-node, Multi-GPUs synchronized training**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8100afe4",
   "metadata": {},
   "source": [
    "### Installation ###\n",
    "+ **Requirements**\n",
    "    - TensorFlow 2.x\n",
    "\n",
    "\n",
    "+ **Get SOK from NGC** <br>\n",
    "The SparseOperationKit is preinstalled in the [Merlin Tensorflow Training Container](https://ngc.nvidia.com/catalog/containers/nvidia:merlin:merlin-tensorflow-training): `nvcr.io/nvidia/merlin/merlin-tensorflow-training:21.12`. <br>\n",
    "You can check the existence of required libraries by running the following Python code after launching this container.\n",
    "```shell\n",
    "$ python3 -c \"import sparse_operation_kit as sok\"\n",
    "```\n",
    "\n",
    "+ **Build SOK from Souce Code** <br>\n",
    "If you want to build SparseOperationKit from the souce code instead of using the NGC container, please refer to the [Setup development environment](../../docs/hugectr_contributor_guide.md#build-sparse-operation-kit-sok-from-source)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "232279bf",
   "metadata": {},
   "source": [
    "### Single-node, Multi-GPUs synchronized training ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9c999d",
   "metadata": {},
   "source": [
    "Firstly, specify hyper parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a543df58",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reset -f\n",
    "\n",
    "args = dict()\n",
    "\n",
    "args[\"gpu_num\"] = 8                               # the number of available GPUs\n",
    "args[\"iter_num\"] = 50                             # the number of training iteration\n",
    "args[\"max_vocabulary_size_per_gpu\"] = 1024\n",
    "args[\"slot_num\"] = 10                             # the number of feature fields in this embedding layer\n",
    "args[\"max_nnz\"] = 4                               # the maximum number of valid features in each slot\n",
    "args[\"embedding_vec_size\"] = 4                    # the dimension of embedding vectors\n",
    "args[\"combiner\"] = \"mean\"                         # the reduction combiner used intra slots, it can be [mean, sum]\n",
    "args[\"global_batch_size\"] = 65536                 # the globally batchsize for all GPUs\n",
    "args[\"optimizer\"] = \"plugin_adam\"                 # the optimizer used for training, it can be [plugin_adam, adam, sgd]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57b5905b",
   "metadata": {},
   "source": [
    "Secondly, import the used modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "75d2fccc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]: sparse_operation_kit is imported\n"
     ]
    }
   ],
   "source": [
    "import sys, os, json\n",
    "import sparse_operation_kit as sok\n",
    "import tensorflow as tf\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"] = \",\".join(map(str, range(args[\"gpu_num\"])))\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "076dd2da",
   "metadata": {},
   "source": [
    "Thirdly, define a DNN model using TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "874566b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TfDemo(tf.keras.models.Model):\n",
    "    def __init__(self, \n",
    "                 init_tensors, \n",
    "                 combiner, \n",
    "                 global_batch_size,\n",
    "                 slot_num, \n",
    "                 embedding_vec_size,\n",
    "                 **kwargs):\n",
    "        super(TfDemo, self).__init__(**kwargs)\n",
    "        self.combiner = combiner\n",
    "        self.global_batch_size = global_batch_size\n",
    "        self.slot_num = slot_num\n",
    "        self.embedding_vec_size = embedding_vec_size\n",
    "\n",
    "        self.init_tensors = init_tensors\n",
    "        self.params = tf.Variable(initial_value=tf.concat(self.init_tensors, axis=0))\n",
    "\n",
    "        self.dense_layer = tf.keras.layers.Dense(units=1, activation=None,\n",
    "                                                 kernel_initializer=\"ones\",\n",
    "                                                 bias_initializer=\"zeros\")\n",
    "\n",
    "    def call(self, inputs, training=True):\n",
    "        # [batchsize * slot_num, embedding_vec_size]\n",
    "        embedding_vector = tf.nn.embedding_lookup_sparse(params=self.params, sp_ids=inputs,\n",
    "                                                        sp_weights=None, combiner=self.combiner)\n",
    "\n",
    "        # [batchsize, slot_num * embedding_vec_size]\n",
    "        embedding_vector = tf.reshape(embedding_vector, shape=[self.global_batch_size, self.slot_num * self.embedding_vec_size])\n",
    "        logit = self.dense_layer(embedding_vector)\n",
    "        return logit, embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d24277",
   "metadata": {},
   "source": [
    "Fourthly, define the same DNN model using SOK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c1ab6d44",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SOKDemo(tf.keras.models.Model):\n",
    "    def __init__(self,\n",
    "                 combiner,\n",
    "                 max_vocabulary_size_per_gpu,\n",
    "                 slot_num,\n",
    "                 max_nnz,\n",
    "                 embedding_vec_size, \n",
    "                 **kwargs):\n",
    "        super(SOKDemo, self).__init__(**kwargs)\n",
    "\n",
    "        self.combiner = combiner\n",
    "        self.max_vocabulary_size_per_gpu = max_vocabulary_size_per_gpu\n",
    "        self.slot_num = slot_num\n",
    "        self.max_nnz = max_nnz\n",
    "        self.embedding_vec_size = embedding_vec_size\n",
    "\n",
    "        self.embedding_layer = sok.DistributedEmbedding(combiner=self.combiner,\n",
    "                                                           max_vocabulary_size_per_gpu=self.max_vocabulary_size_per_gpu,\n",
    "                                                           embedding_vec_size=self.embedding_vec_size,\n",
    "                                                           slot_num=self.slot_num,\n",
    "                                                           max_nnz=self.max_nnz)\n",
    "\n",
    "        self.dense_layer = tf.keras.layers.Dense(units=1, activation=None,\n",
    "                                                 kernel_initializer=\"ones\",\n",
    "                                                 bias_initializer=\"zeros\")\n",
    "\n",
    "    def call(self, inputs, training=True):\n",
    "        # [batchsize, slot_num, embedding_vec_size]\n",
    "        embedding_vector = self.embedding_layer(inputs, training=training)\n",
    "        # [batchsize, slot_num * embedding_vec_size]\n",
    "        embedding_vector = tf.reshape(embedding_vector, shape=[-1, self.slot_num * self.embedding_vec_size])\n",
    "        # [batchsize, 1]\n",
    "        logit = self.dense_layer(embedding_vector)\n",
    "        return logit, embedding_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c410690c",
   "metadata": {},
   "source": [
    "Fifthly, generate synthetic dataset and initial values that is used to initialize embedding parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a588e648",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import utility python script\n",
    "sys.path.append(\"../unit_test/test_scripts/tf2/\")\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f2a621c4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]: begin to generate random samples\n",
      "[INFO]: generated random samples\n"
     ]
    }
   ],
   "source": [
    "# -1 is used to represent the invalid keys\n",
    "random_samples = utils.generate_random_samples(num_of_samples=args[\"global_batch_size\"] * args[\"iter_num\"],\n",
    "                                               vocabulary_size=args[\"gpu_num\"] * args[\"max_vocabulary_size_per_gpu\"],\n",
    "                                               slot_num=args[\"slot_num\"],\n",
    "                                               max_nnz=args[\"max_nnz\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "df700223",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[[ 237,   38,   -1,   -1],\n",
       "         [1255,  921,   -1,   -1],\n",
       "         [2139,   -1,   -1,   -1],\n",
       "         ...,\n",
       "         [  -1,   -1,   -1,   -1],\n",
       "         [7018, 6987,   -1,   -1],\n",
       "         [7875,   -1,   -1,   -1]],\n",
       " \n",
       "        [[ 749,  718,  680,  642],\n",
       "         [1606,  894,   -1,   -1],\n",
       "         [1859, 1782,   -1,   -1],\n",
       "         ...,\n",
       "         [6455, 6384, 6321,   -1],\n",
       "         [6589,   -1,   -1,   -1],\n",
       "         [7890,   -1,   -1,   -1]],\n",
       " \n",
       "        [[ 653,  582,   -1,   -1],\n",
       "         [1063,  929,  858,   -1],\n",
       "         [1953,   -1,   -1,   -1],\n",
       "         ...,\n",
       "         [6209, 6084, 5866, 5741],\n",
       "         [6942,   -1,   -1,   -1],\n",
       "         [7886, 7410,   -1,   -1]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 166,   -1,   -1,   -1],\n",
       "         [1580, 1210, 1058,  913],\n",
       "         [2326, 2174, 1796,   -1],\n",
       "         ...,\n",
       "         [5917,   -1,   -1,   -1],\n",
       "         [7227, 7066, 6738,   -1],\n",
       "         [8056,   -1,   -1,   -1]],\n",
       " \n",
       "        [[ 356,   -1,   -1,   -1],\n",
       "         [1346, 1177,   -1,   -1],\n",
       "         [2005, 1675,   -1,   -1],\n",
       "         ...,\n",
       "         [6320, 6285,   -1,   -1],\n",
       "         [7014, 6952,   -1,   -1],\n",
       "         [7736, 7709, 7647,   -1]],\n",
       " \n",
       "        [[ 152,   -1,   -1,   -1],\n",
       "         [ 936,  874,  839,   -1],\n",
       "         [2446, 2393, 2006, 1953],\n",
       "         ...,\n",
       "         [6243, 5750,   -1,   -1],\n",
       "         [6675,   -1,   -1,   -1],\n",
       "         [8084, 8039, 7652,   -1]]]),\n",
       " array([[0],\n",
       "        [0],\n",
       "        [1],\n",
       "        ...,\n",
       "        [0],\n",
       "        [1],\n",
       "        [1]]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check ramdom_samples\n",
    "random_samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "de088175",
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate initial value for embedding parameters\n",
    "init_tensors = utils.get_ones_tensor(max_vocab_size_per_gpu=args[\"max_vocabulary_size_per_gpu\"],\n",
    "                                     embedding_vec_size=args[\"embedding_vec_size\"],\n",
    "                                     num=args[\"gpu_num\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6e1f0b5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32),\n",
       " array([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        ...,\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]], dtype=float32)]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check init_tensors\n",
    "init_tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdc0335b",
   "metadata": {},
   "source": [
    "Sixly, define training loop for TensorFlow and SOK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "bdb376ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_tf_demo(args, init_tensors, *random_samples):\n",
    "    dataset = utils.tf_dataset(*random_samples, batchsize=args[\"global_batch_size\"], to_sparse_tensor=True, repeat=1)\n",
    "\n",
    "    loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
    "\n",
    "    tf_demo = TfDemo(init_tensors, args[\"combiner\"], args[\"global_batch_size\"], \n",
    "                     args[\"slot_num\"], args[\"embedding_vec_size\"])\n",
    "\n",
    "    optimizer = utils.get_dense_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "\n",
    "    @tf.function\n",
    "    def _train_step(inputs, labels):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logit, embedding_vector = tf_demo(inputs, training=True)\n",
    "            loss = loss_fn(labels, logit)\n",
    "        grads = tape.gradient(loss, tf_demo.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, tf_demo.trainable_variables))\n",
    "        return logit, embedding_vector\n",
    "\n",
    "    tf_results = list()\n",
    "\n",
    "    for i, (sparse_tensors, labels) in enumerate(dataset):\n",
    "        print(\"-\"*30, str(i), \"-\"*30)\n",
    "        logit, embedding_vector = _train_step(sparse_tensors, labels)\n",
    "        print(\"[INFO]: embedding_vector:\\n\", embedding_vector)\n",
    "        tf_results.append(embedding_vector)\n",
    "\n",
    "        # FIXME: because plugin sleepd, here is only used for \n",
    "        # simulate the same DNN structure. \n",
    "        import time\n",
    "        time.sleep(0.2) # seconds\n",
    "\n",
    "    return tf_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cbe95077",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_sok_demo(args, init_tensors, *random_samples):\n",
    "    strategy = tf.distribute.MirroredStrategy()\n",
    "    with strategy.scope():\n",
    "        result = sok.Init(global_batch_size=args[\"global_batch_size\"])\n",
    "\n",
    "        plugin_demo = SOKDemo(combiner=args[\"combiner\"], \n",
    "                                 max_vocabulary_size_per_gpu=args[\"max_vocabulary_size_per_gpu\"],\n",
    "                                 slot_num=args[\"slot_num\"], max_nnz=args[\"max_nnz\"],\n",
    "                                 embedding_vec_size=args[\"embedding_vec_size\"])\n",
    "\n",
    "        emb_opt = utils.get_embedding_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "        dense_opt = utils.get_dense_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "\n",
    "    plugin_saver = sok.Saver()\n",
    "\n",
    "    plugin_saver.load_embedding_values(plugin_demo.embedding_layer.embedding_variable, init_tensors)\n",
    "\n",
    "    loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)\n",
    "    def _replica_loss(labels, logits):\n",
    "        loss = loss_fn(labels, logits)\n",
    "        return tf.nn.compute_average_loss(loss, global_batch_size=args[\"global_batch_size\"])\n",
    "\n",
    "    @tf.function\n",
    "    def _train_step(inputs, labels):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logit, embedding_vector = plugin_demo(inputs, training=True)\n",
    "            loss = _replica_loss(labels, logit)\n",
    "        embedding_variables, other_variable = sok.split_embedding_variable_from_others(plugin_demo.trainable_variables)\n",
    "        grads, emb_grads = tape.gradient(loss, [other_variable, embedding_variables])\n",
    "        if 'plugin' not in args[\"optimizer\"]:\n",
    "            with sok.OptimizerScope(embedding_variables):\n",
    "                emb_opt.apply_gradients(zip(emb_grads, embedding_variables),\n",
    "                                        experimental_aggregate_gradients=False)\n",
    "        else:\n",
    "            emb_opt.apply_gradients(zip(emb_grads, embedding_variables),\n",
    "                                    experimental_aggregate_gradients=False)\n",
    "        dense_opt.apply_gradients(zip(grads, other_variable))\n",
    "        return logit, embedding_vector\n",
    "\n",
    "    sok_results = list()\n",
    "\n",
    "    def _dataset_fn(input_context):\n",
    "        replica_batch_size = input_context.get_per_replica_batch_size(args[\"global_batch_size\"])\n",
    "        dataset = utils.tf_dataset(*random_samples, batchsize=replica_batch_size, to_sparse_tensor=True, repeat=1)\n",
    "        dataset = dataset.shard(input_context.num_input_pipelines, input_context.input_pipeline_id)\n",
    "        return dataset\n",
    "\n",
    "    dataset = strategy.distribute_datasets_from_function(_dataset_fn)\n",
    "    \n",
    "    for i, (sparse_tensors, replica_labels) in enumerate(dataset):\n",
    "        print(\"-\" * 30, \"step \", str(i), \"-\" * 30)\n",
    "        logit, embedding_vector = strategy.run(_train_step, args=(sparse_tensors, replica_labels))\n",
    "        print(\"[INFO]: embedding_vector\\n\", embedding_vector)\n",
    "        sok_results.append(embedding_vector)\n",
    "\n",
    "        # FIXME: when the forward computation is too fast, there\n",
    "        # may exist some conficts with datareader, which cause the program hang.\n",
    "        import time\n",
    "        time.sleep(0.2) # seconds\n",
    "\n",
    "    return sok_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d560ad5",
   "metadata": {},
   "source": [
    "Sevenly, start training process. Compare whether the embedding vectors obtained from TensorFlow and SOK are consistent in all iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e1e4cb2a",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-12-07 04:48:26.388375: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.388450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13631 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.390576: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.390609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14631 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:07:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.392713: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.392745: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14631 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.394732: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.394765: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14631 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0b:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.396761: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.396793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:4 with 14631 MB memory:  -> device: 4, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.398758: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.398790: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:5 with 14631 MB memory:  -> device: 5, name: Tesla V100-SXM2-16GB, pci bus id: 0000:86:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.400730: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.400763: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:6 with 14631 MB memory:  -> device: 6, name: Tesla V100-SXM2-16GB, pci bus id: 0000:89:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:26.402711: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:48:26.402743: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:7 with 14631 MB memory:  -> device: 7, name: Tesla V100-SXM2-16GB, pci bus id: 0000:8a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:48:29.392137: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 0 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 1 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.9005389  0.9005389  0.9005389  ... 0.9004773  0.9004773  0.9004773 ]\n",
      " [0.9004776  0.9004776  0.9004776  ... 0.90067196 0.90067196 0.90067196]\n",
      " [0.90046406 0.90046406 0.90046406 ... 0.9004576  0.9004576  0.9004576 ]\n",
      " ...\n",
      " [0.9005024  0.9005024  0.9005024  ... 0.9005311  0.9005311  0.9005311 ]\n",
      " [0.9005841  0.9005841  0.9005841  ... 0.90050334 0.90050334 0.90050334]\n",
      " [0.90044    0.90044    0.90044    ... 0.90049696 0.90049696 0.90049696]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 2 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.8012184  0.8012184  0.8012184  ... 0.80078053 0.80078053 0.80078053]\n",
      " [0.8010931  0.8010931  0.8010931  ... 0.8017576  0.8017576  0.8017576 ]\n",
      " [0.8011229  0.8011229  0.8011229  ... 0.8008481  0.8008481  0.8008481 ]\n",
      " ...\n",
      " [0.8013132  0.8013132  0.8013132  ... 0.8011102  0.8011102  0.8011102 ]\n",
      " [0.80126804 0.80126804 0.80126804 ... 0.8013719  0.8013719  0.8013719 ]\n",
      " [0.80104816 0.80104816 0.80104816 ... 0.803952   0.803952   0.803952  ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 3 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.7042218  0.7042218  0.7042218  ... 0.7051984  0.7051984  0.7051984 ]\n",
      " [0.7036438  0.7036438  0.7036438  ... 0.7032419  0.7032419  0.7032419 ]\n",
      " [0.70129055 0.70129055 0.70129055 ... 0.70223    0.70223    0.70223   ]\n",
      " ...\n",
      " [0.70187265 0.70187265 0.70187265 ... 0.7067666  0.7067666  0.7067666 ]\n",
      " [0.7038647  0.7038647  0.7038647  ... 0.70275354 0.70275354 0.70275354]\n",
      " [0.7041642  0.7041642  0.7041642  ... 0.70362234 0.70362234 0.70362234]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 4 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.60573584 0.60573584 0.60573584 ... 0.6049477  0.6049477  0.6049477 ]\n",
      " [0.6066303  0.6066303  0.6066303  ... 0.6060393  0.6060393  0.6060393 ]\n",
      " [0.6072234  0.6072234  0.6072234  ... 0.60453373 0.60453373 0.60453373]\n",
      " ...\n",
      " [0.60645366 0.60645366 0.60645366 ... 0.60563123 0.60563123 0.60563123]\n",
      " [0.6045116  0.6045116  0.6045116  ... 0.60481894 0.60481894 0.60481894]\n",
      " [0.60465115 0.60465115 0.60465115 ... 0.6037544  0.6037544  0.6037544 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 5 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.51534414 0.51534414 0.51534414 ... 0.5110701  0.5110701  0.5110701 ]\n",
      " [0.5077944  0.5077944  0.5077944  ... 0.5091696  0.5091696  0.5091696 ]\n",
      " [0.51320696 0.51320696 0.51320696 ... 0.51868916 0.51868916 0.51868916]\n",
      " ...\n",
      " [0.512895   0.512895   0.512895   ... 0.51354986 0.51354986 0.51354986]\n",
      " [0.5107079  0.5107079  0.5107079  ... 0.5182532  0.5182532  0.5182532 ]\n",
      " [0.5100364  0.5100364  0.5100364  ... 0.5142635  0.5142635  0.5142635 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 6 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.4215302  0.4215302  0.4215302  ... 0.42518848 0.42518848 0.42518848]\n",
      " [0.41708758 0.41708758 0.41708758 ... 0.41547316 0.41547316 0.41547316]\n",
      " [0.41185933 0.41185933 0.41185933 ... 0.41378838 0.41378838 0.41378838]\n",
      " ...\n",
      " [0.41671106 0.41671106 0.41671106 ... 0.40972468 0.40972468 0.40972468]\n",
      " [0.4155799  0.4155799  0.4155799  ... 0.41866857 0.41866857 0.41866857]\n",
      " [0.42025787 0.42025787 0.42025787 ... 0.41543674 0.41543674 0.41543674]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 7 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.3316445  0.3316445  0.3316445  ... 0.33699805 0.33699805 0.33699805]\n",
      " [0.32833043 0.32833043 0.32833043 ... 0.3246198  0.3246198  0.3246198 ]\n",
      " [0.32871115 0.32871115 0.32871115 ... 0.32886964 0.32886964 0.32886964]\n",
      " ...\n",
      " [0.32005632 0.32005632 0.32005632 ... 0.33087358 0.33087358 0.33087358]\n",
      " [0.32934743 0.32934743 0.32934743 ... 0.32109705 0.32109705 0.32109705]\n",
      " [0.32427126 0.32427126 0.32427126 ... 0.3273436  0.3273436  0.3273436 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 8 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.24441278 0.24441278 0.24441278 ... 0.24178444 0.24178444 0.24178444]\n",
      " [0.2394062  0.2394062  0.2394062  ... 0.24691832 0.24691832 0.24691832]\n",
      " [0.24403517 0.24403517 0.24403517 ... 0.24393018 0.24393018 0.24393018]\n",
      " ...\n",
      " [0.24793357 0.24793357 0.24793357 ... 0.23112187 0.23112187 0.23112187]\n",
      " [0.24537155 0.24537155 0.24537155 ... 0.24710768 0.24710768 0.24710768]\n",
      " [0.24537086 0.24537086 0.24537086 ... 0.23915745 0.23915745 0.23915745]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 9 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.1615668  0.1615668  0.1615668  ... 0.16512388 0.16512388 0.16512388]\n",
      " [0.1707711  0.1707711  0.1707711  ... 0.16517943 0.16517943 0.16517943]\n",
      " [0.16769373 0.16769373 0.16769373 ... 0.16542022 0.16542022 0.16542022]\n",
      " ...\n",
      " [0.16749145 0.16749145 0.16749145 ... 0.15404093 0.15404093 0.15404093]\n",
      " [0.16487949 0.16487949 0.16487949 ... 0.15122642 0.15122642 0.15122642]\n",
      " [0.16254517 0.16254517 0.16254517 ... 0.1469526  0.1469526  0.1469526 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 10 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.10003155 0.10003155 0.10003155 ... 0.08497272 0.08497272 0.08497272]\n",
      " [0.09688364 0.09688364 0.09688364 ... 0.0960445  0.0960445  0.0960445 ]\n",
      " [0.12318983 0.12318983 0.12318983 ... 0.09761778 0.09761778 0.09761778]\n",
      " ...\n",
      " [0.09408723 0.09408723 0.09408723 ... 0.09515518 0.09515518 0.09515518]\n",
      " [0.08360902 0.08360902 0.08360902 ... 0.09440356 0.09440356 0.09440356]\n",
      " [0.08900392 0.08900392 0.08900392 ... 0.09011607 0.09011607 0.09011607]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 11 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.02892517 0.02892517 0.02892517 ... 0.03236027 0.03236027 0.03236027]\n",
      " [0.02894042 0.02894042 0.02894042 ... 0.02948065 0.02948065 0.02948065]\n",
      " [0.03007229 0.03007229 0.03007229 ... 0.03537836 0.03537836 0.03537836]\n",
      " ...\n",
      " [0.02920975 0.02920975 0.02920975 ... 0.03624295 0.03624295 0.03624295]\n",
      " [0.02694957 0.02694957 0.02694957 ... 0.03740635 0.03740635 0.03740635]\n",
      " [0.03670081 0.03670081 0.03670081 ... 0.03013404 0.03013404 0.03013404]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 12 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.02506138 -0.02506138 -0.02506138 ... -0.02554815 -0.02554815\n",
      "  -0.02554815]\n",
      " [-0.02334073 -0.02334073 -0.02334073 ... -0.03851846 -0.03851846\n",
      "  -0.03851846]\n",
      " [-0.03781048 -0.03781048 -0.03781048 ... -0.03588433 -0.03588433\n",
      "  -0.03588433]\n",
      " ...\n",
      " [-0.03608987 -0.03608987 -0.03608987 ... -0.02981279 -0.02981279\n",
      "  -0.02981279]\n",
      " [-0.02650625 -0.02650625 -0.02650625 ... -0.02783397 -0.02783397\n",
      "  -0.02783397]\n",
      " [-0.03981005 -0.03981005 -0.03981005 ... -0.01936699 -0.01936699\n",
      "  -0.01936699]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 13 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.08360223 -0.08360223 -0.08360223 ... -0.08796514 -0.08796514\n",
      "  -0.08796514]\n",
      " [-0.09025344 -0.09025344 -0.09025344 ... -0.05054506 -0.05054506\n",
      "  -0.05054506]\n",
      " [-0.07120632 -0.07120632 -0.07120632 ... -0.08376878 -0.08376878\n",
      "  -0.08376878]\n",
      " ...\n",
      " [-0.10011729 -0.10011729 -0.10011729 ... -0.09384847 -0.09384847\n",
      "  -0.09384847]\n",
      " [-0.09645236 -0.09645236 -0.09645236 ... -0.08332622 -0.08332622\n",
      "  -0.08332622]\n",
      " [-0.08130629 -0.08130629 -0.08130629 ... -0.08048826 -0.08048826\n",
      "  -0.08048826]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 14 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13518712 -0.13518712 -0.13518712 ... -0.12721723 -0.12721723\n",
      "  -0.12721723]\n",
      " [-0.1107835  -0.1107835  -0.1107835  ... -0.1284406  -0.1284406\n",
      "  -0.1284406 ]\n",
      " [-0.10312327 -0.10312327 -0.10312327 ... -0.13691477 -0.13691477\n",
      "  -0.13691477]\n",
      " ...\n",
      " [-0.11958332 -0.11958332 -0.11958332 ... -0.10025437 -0.10025437\n",
      "  -0.10025437]\n",
      " [-0.10139414 -0.10139414 -0.10139414 ... -0.12497405 -0.12497405\n",
      "  -0.12497405]\n",
      " [-0.12762044 -0.12762044 -0.12762044 ... -0.11974303 -0.11974303\n",
      "  -0.11974303]], shape=(65536, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 15 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15322745 -0.15322745 -0.15322745 ... -0.17870769 -0.17870769\n",
      "  -0.17870769]\n",
      " [-0.15844432 -0.15844432 -0.15844432 ... -0.17112882 -0.17112882\n",
      "  -0.17112882]\n",
      " [-0.14272878 -0.14272878 -0.14272878 ... -0.18848768 -0.18848768\n",
      "  -0.18848768]\n",
      " ...\n",
      " [-0.17903242 -0.17903242 -0.17903242 ... -0.21368095 -0.21368095\n",
      "  -0.21368095]\n",
      " [-0.18603516 -0.18603516 -0.18603516 ... -0.18102147 -0.18102147\n",
      "  -0.18102147]\n",
      " [-0.16306126 -0.16306126 -0.16306126 ... -0.18238421 -0.18238421\n",
      "  -0.18238421]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 16 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22411308 -0.22411308 -0.22411308 ... -0.19928825 -0.19928825\n",
      "  -0.19928825]\n",
      " [-0.20470376 -0.20470376 -0.20470376 ... -0.19559422 -0.19559422\n",
      "  -0.19559422]\n",
      " [-0.19532926 -0.19532926 -0.19532926 ... -0.22929913 -0.22929913\n",
      "  -0.22929913]\n",
      " ...\n",
      " [-0.21242538 -0.21242538 -0.21242538 ... -0.20085888 -0.20085888\n",
      "  -0.20085888]\n",
      " [-0.1980874  -0.1980874  -0.1980874  ... -0.21918803 -0.21918803\n",
      "  -0.21918803]\n",
      " [-0.24142903 -0.24142903 -0.24142903 ... -0.21955004 -0.21955004\n",
      "  -0.21955004]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 17 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23744297 -0.23744297 -0.23744297 ... -0.25772938 -0.25772938\n",
      "  -0.25772938]\n",
      " [-0.23552291 -0.23552291 -0.23552291 ... -0.2481287  -0.2481287\n",
      "  -0.2481287 ]\n",
      " [-0.2525279  -0.2525279  -0.2525279  ... -0.24318762 -0.24318762\n",
      "  -0.24318762]\n",
      " ...\n",
      " [-0.24147987 -0.24147987 -0.24147987 ... -0.245133   -0.245133\n",
      "  -0.245133  ]\n",
      " [-0.21726954 -0.21726954 -0.21726954 ... -0.24497202 -0.24497202\n",
      "  -0.24497202]\n",
      " [-0.23562382 -0.23562382 -0.23562382 ... -0.2501109  -0.2501109\n",
      "  -0.2501109 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 18 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.28225696 -0.28225696 -0.28225696 ... -0.26526678 -0.26526678\n",
      "  -0.26526678]\n",
      " [-0.29595616 -0.29595616 -0.29595616 ... -0.24435407 -0.24435407\n",
      "  -0.24435407]\n",
      " [-0.27789468 -0.27789468 -0.27789468 ... -0.2857512  -0.2857512\n",
      "  -0.2857512 ]\n",
      " ...\n",
      " [-0.29183844 -0.29183844 -0.29183844 ... -0.29049727 -0.29049727\n",
      "  -0.29049727]\n",
      " [-0.27364194 -0.27364194 -0.27364194 ... -0.26075423 -0.26075423\n",
      "  -0.26075423]\n",
      " [-0.23404743 -0.23404743 -0.23404743 ... -0.27080432 -0.27080432\n",
      "  -0.27080432]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 19 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.27474    -0.27474    -0.27474    ... -0.26349118 -0.26349118\n",
      "  -0.26349118]\n",
      " [-0.28046262 -0.28046262 -0.28046262 ... -0.28547356 -0.28547356\n",
      "  -0.28547356]\n",
      " [-0.2812343  -0.2812343  -0.2812343  ... -0.31628248 -0.31628248\n",
      "  -0.31628248]\n",
      " ...\n",
      " [-0.31159952 -0.31159952 -0.31159952 ... -0.27967572 -0.27967572\n",
      "  -0.27967572]\n",
      " [-0.2989855  -0.2989855  -0.2989855  ... -0.29216075 -0.29216075\n",
      "  -0.29216075]\n",
      " [-0.2977442  -0.2977442  -0.2977442  ... -0.2908248  -0.2908248\n",
      "  -0.2908248 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 20 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.30255374 -0.30255374 -0.30255374 ... -0.28498274 -0.28498274\n",
      "  -0.28498274]\n",
      " [-0.2982529  -0.2982529  -0.2982529  ... -0.3098672  -0.3098672\n",
      "  -0.3098672 ]\n",
      " [-0.29218975 -0.29218975 -0.29218975 ... -0.30057096 -0.30057096\n",
      "  -0.30057096]\n",
      " ...\n",
      " [-0.31813034 -0.31813034 -0.31813034 ... -0.2846597  -0.2846597\n",
      "  -0.2846597 ]\n",
      " [-0.31135598 -0.31135598 -0.31135598 ... -0.29507133 -0.29507133\n",
      "  -0.29507133]\n",
      " [-0.29123628 -0.29123628 -0.29123628 ... -0.30289805 -0.30289805\n",
      "  -0.30289805]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 21 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.31248215 -0.31248215 -0.31248215 ... -0.31547493 -0.31547493\n",
      "  -0.31547493]\n",
      " [-0.29827893 -0.29827893 -0.29827893 ... -0.32298163 -0.32298163\n",
      "  -0.32298163]\n",
      " [-0.30721414 -0.30721414 -0.30721414 ... -0.3031354  -0.3031354\n",
      "  -0.3031354 ]\n",
      " ...\n",
      " [-0.32914197 -0.32914197 -0.32914197 ... -0.30411258 -0.30411258\n",
      "  -0.30411258]\n",
      " [-0.29133913 -0.29133913 -0.29133913 ... -0.2952198  -0.2952198\n",
      "  -0.2952198 ]\n",
      " [-0.3171327  -0.3171327  -0.3171327  ... -0.2892619  -0.2892619\n",
      "  -0.2892619 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 22 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.3002741  -0.3002741  -0.3002741  ... -0.32113612 -0.32113612\n",
      "  -0.32113612]\n",
      " [-0.31446022 -0.31446022 -0.31446022 ... -0.3182202  -0.3182202\n",
      "  -0.3182202 ]\n",
      " [-0.28149402 -0.28149402 -0.28149402 ... -0.30891025 -0.30891025\n",
      "  -0.30891025]\n",
      " ...\n",
      " [-0.2992421  -0.2992421  -0.2992421  ... -0.29946035 -0.29946035\n",
      "  -0.29946035]\n",
      " [-0.33087903 -0.33087903 -0.33087903 ... -0.29013512 -0.29013512\n",
      "  -0.29013512]\n",
      " [-0.2990111  -0.2990111  -0.2990111  ... -0.30843148 -0.30843148\n",
      "  -0.30843148]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 23 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.29579163 -0.29579163 -0.29579163 ... -0.29633436 -0.29633436\n",
      "  -0.29633436]\n",
      " [-0.2936103  -0.2936103  -0.2936103  ... -0.26991478 -0.26991478\n",
      "  -0.26991478]\n",
      " [-0.2689963  -0.2689963  -0.2689963  ... -0.29569775 -0.29569775\n",
      "  -0.29569775]\n",
      " ...\n",
      " [-0.28067687 -0.28067687 -0.28067687 ... -0.2907971  -0.2907971\n",
      "  -0.2907971 ]\n",
      " [-0.3203141  -0.3203141  -0.3203141  ... -0.2895282  -0.2895282\n",
      "  -0.2895282 ]\n",
      " [-0.29284713 -0.29284713 -0.29284713 ... -0.308792   -0.308792\n",
      "  -0.308792  ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 24 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.26862147 -0.26862147 -0.26862147 ... -0.2935341  -0.2935341\n",
      "  -0.2935341 ]\n",
      " [-0.27740854 -0.27740854 -0.27740854 ... -0.25745258 -0.25745258\n",
      "  -0.25745258]\n",
      " [-0.2925211  -0.2925211  -0.2925211  ... -0.30540887 -0.30540887\n",
      "  -0.30540887]\n",
      " ...\n",
      " [-0.2778787  -0.2778787  -0.2778787  ... -0.27529776 -0.27529776\n",
      "  -0.27529776]\n",
      " [-0.2766596  -0.2766596  -0.2766596  ... -0.27657467 -0.27657467\n",
      "  -0.27657467]\n",
      " [-0.26452947 -0.26452947 -0.26452947 ... -0.25722316 -0.25722316\n",
      "  -0.25722316]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 25 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.2711467  -0.2711467  -0.2711467  ... -0.2525984  -0.2525984\n",
      "  -0.2525984 ]\n",
      " [-0.26358426 -0.26358426 -0.26358426 ... -0.24389586 -0.24389586\n",
      "  -0.24389586]\n",
      " [-0.2759707  -0.2759707  -0.2759707  ... -0.24856615 -0.24856615\n",
      "  -0.24856615]\n",
      " ...\n",
      " [-0.29495502 -0.29495502 -0.29495502 ... -0.24477978 -0.24477978\n",
      "  -0.24477978]\n",
      " [-0.23777947 -0.23777947 -0.23777947 ... -0.27463824 -0.27463824\n",
      "  -0.27463824]\n",
      " [-0.27357185 -0.27357185 -0.27357185 ... -0.2804342  -0.2804342\n",
      "  -0.2804342 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 26 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.21731079 -0.21731079 -0.21731079 ... -0.23799147 -0.23799147\n",
      "  -0.23799147]\n",
      " [-0.19461295 -0.19461295 -0.19461295 ... -0.24835142 -0.24835142\n",
      "  -0.24835142]\n",
      " [-0.25106174 -0.25106174 -0.25106174 ... -0.22300126 -0.22300126\n",
      "  -0.22300126]\n",
      " ...\n",
      " [-0.24908134 -0.24908134 -0.24908134 ... -0.22033095 -0.22033095\n",
      "  -0.22033095]\n",
      " [-0.22769272 -0.22769272 -0.22769272 ... -0.20756304 -0.20756304\n",
      "  -0.20756304]\n",
      " [-0.25136542 -0.25136542 -0.25136542 ... -0.2374534  -0.2374534\n",
      "  -0.2374534 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 27 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20556271 -0.20556271 -0.20556271 ... -0.19028768 -0.19028768\n",
      "  -0.19028768]\n",
      " [-0.22185035 -0.22185035 -0.22185035 ... -0.22860134 -0.22860134\n",
      "  -0.22860134]\n",
      " [-0.2227293  -0.2227293  -0.2227293  ... -0.22487032 -0.22487032\n",
      "  -0.22487032]\n",
      " ...\n",
      " [-0.2157962  -0.2157962  -0.2157962  ... -0.18616611 -0.18616611\n",
      "  -0.18616611]\n",
      " [-0.18807095 -0.18807095 -0.18807095 ... -0.20386793 -0.20386793\n",
      "  -0.20386793]\n",
      " [-0.19489944 -0.19489944 -0.19489944 ... -0.24171107 -0.24171107\n",
      "  -0.24171107]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 28 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1896869  -0.1896869  -0.1896869  ... -0.1767119  -0.1767119\n",
      "  -0.1767119 ]\n",
      " [-0.21345438 -0.21345438 -0.21345438 ... -0.18973006 -0.18973006\n",
      "  -0.18973006]\n",
      " [-0.19733995 -0.19733995 -0.19733995 ... -0.17629492 -0.17629492\n",
      "  -0.17629492]\n",
      " ...\n",
      " [-0.21840473 -0.21840473 -0.21840473 ... -0.19216579 -0.19216579\n",
      "  -0.19216579]\n",
      " [-0.21679032 -0.21679032 -0.21679032 ... -0.21031338 -0.21031338\n",
      "  -0.21031338]\n",
      " [-0.17250302 -0.17250302 -0.17250302 ... -0.17423418 -0.17423418\n",
      "  -0.17423418]], shape=(65536, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 29 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15862423 -0.15862423 -0.15862423 ... -0.17314598 -0.17314598\n",
      "  -0.17314598]\n",
      " [-0.1752002  -0.1752002  -0.1752002  ... -0.17943908 -0.17943908\n",
      "  -0.17943908]\n",
      " [-0.20004535 -0.20004535 -0.20004535 ... -0.15819792 -0.15819792\n",
      "  -0.15819792]\n",
      " ...\n",
      " [-0.17449039 -0.17449039 -0.17449039 ... -0.17067263 -0.17067263\n",
      "  -0.17067263]\n",
      " [-0.19266994 -0.19266994 -0.19266994 ... -0.17156011 -0.17156011\n",
      "  -0.17156011]\n",
      " [-0.20928861 -0.20928861 -0.20928861 ... -0.13899939 -0.13899939\n",
      "  -0.13899939]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 30 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.17543076 -0.17543076 -0.17543076 ... -0.1694403  -0.1694403\n",
      "  -0.1694403 ]\n",
      " [-0.17624065 -0.17624065 -0.17624065 ... -0.19092642 -0.19092642\n",
      "  -0.19092642]\n",
      " [-0.17132854 -0.17132854 -0.17132854 ... -0.15731235 -0.15731235\n",
      "  -0.15731235]\n",
      " ...\n",
      " [-0.15872258 -0.15872258 -0.15872258 ... -0.22980253 -0.22980253\n",
      "  -0.22980253]\n",
      " [-0.17501907 -0.17501907 -0.17501907 ... -0.18760996 -0.18760996\n",
      "  -0.18760996]\n",
      " [-0.18663447 -0.18663447 -0.18663447 ... -0.14019679 -0.14019679\n",
      "  -0.14019679]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 31 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13489929 -0.13489929 -0.13489929 ... -0.14705525 -0.14705525\n",
      "  -0.14705525]\n",
      " [-0.18425956 -0.18425956 -0.18425956 ... -0.16823839 -0.16823839\n",
      "  -0.16823839]\n",
      " [-0.15208828 -0.15208828 -0.15208828 ... -0.14765924 -0.14765924\n",
      "  -0.14765924]\n",
      " ...\n",
      " [-0.11967277 -0.11967277 -0.11967277 ... -0.13228464 -0.13228464\n",
      "  -0.13228464]\n",
      " [-0.15966411 -0.15966411 -0.15966411 ... -0.15740865 -0.15740865\n",
      "  -0.15740865]\n",
      " [-0.18821044 -0.18821044 -0.18821044 ... -0.13960148 -0.13960148\n",
      "  -0.13960148]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 32 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15153545 -0.15153545 -0.15153545 ... -0.14543656 -0.14543656\n",
      "  -0.14543656]\n",
      " [-0.11195644 -0.11195644 -0.11195644 ... -0.15332255 -0.15332255\n",
      "  -0.15332255]\n",
      " [-0.13847102 -0.13847102 -0.13847102 ... -0.15363078 -0.15363078\n",
      "  -0.15363078]\n",
      " ...\n",
      " [-0.1414602  -0.1414602  -0.1414602  ... -0.12048749 -0.12048749\n",
      "  -0.12048749]\n",
      " [-0.14037636 -0.14037636 -0.14037636 ... -0.1324393  -0.1324393\n",
      "  -0.1324393 ]\n",
      " [-0.13409106 -0.13409106 -0.13409106 ... -0.13995743 -0.13995743\n",
      "  -0.13995743]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 33 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13815242 -0.13815242 -0.13815242 ... -0.11207544 -0.11207544\n",
      "  -0.11207544]\n",
      " [-0.1223916  -0.1223916  -0.1223916  ... -0.1228627  -0.1228627\n",
      "  -0.1228627 ]\n",
      " [-0.14009932 -0.14009932 -0.14009932 ... -0.10607965 -0.10607965\n",
      "  -0.10607965]\n",
      " ...\n",
      " [-0.13617386 -0.13617386 -0.13617386 ... -0.1383233  -0.1383233\n",
      "  -0.1383233 ]\n",
      " [-0.09201555 -0.09201555 -0.09201555 ... -0.1556828  -0.1556828\n",
      "  -0.1556828 ]\n",
      " [-0.10313062 -0.10313062 -0.10313062 ... -0.13685633 -0.13685633\n",
      "  -0.13685633]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 34 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13054131 -0.13054131 -0.13054131 ... -0.16830756 -0.16830756\n",
      "  -0.16830756]\n",
      " [-0.12268586 -0.12268586 -0.12268586 ... -0.13875635 -0.13875635\n",
      "  -0.13875635]\n",
      " [-0.15152153 -0.15152153 -0.15152153 ... -0.09778836 -0.09778836\n",
      "  -0.09778836]\n",
      " ...\n",
      " [-0.15402305 -0.15402305 -0.15402305 ... -0.16861378 -0.16861378\n",
      "  -0.16861378]\n",
      " [-0.1669041  -0.1669041  -0.1669041  ... -0.1523404  -0.1523404\n",
      "  -0.1523404 ]\n",
      " [-0.15300009 -0.15300009 -0.15300009 ... -0.13713121 -0.13713121\n",
      "  -0.13713121]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 35 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.127472   -0.127472   -0.127472   ... -0.1591565  -0.1591565\n",
      "  -0.1591565 ]\n",
      " [-0.12941897 -0.12941897 -0.12941897 ... -0.13059841 -0.13059841\n",
      "  -0.13059841]\n",
      " [-0.125127   -0.125127   -0.125127   ... -0.157533   -0.157533\n",
      "  -0.157533  ]\n",
      " ...\n",
      " [-0.13563237 -0.13563237 -0.13563237 ... -0.14324194 -0.14324194\n",
      "  -0.14324194]\n",
      " [-0.12998156 -0.12998156 -0.12998156 ... -0.11999333 -0.11999333\n",
      "  -0.11999333]\n",
      " [-0.15612997 -0.15612997 -0.15612997 ... -0.09071824 -0.09071824\n",
      "  -0.09071824]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 36 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.06406118 -0.06406118 -0.06406118 ... -0.13684662 -0.13684662\n",
      "  -0.13684662]\n",
      " [-0.12287654 -0.12287654 -0.12287654 ... -0.10787309 -0.10787309\n",
      "  -0.10787309]\n",
      " [-0.14317237 -0.14317237 -0.14317237 ... -0.12778284 -0.12778284\n",
      "  -0.12778284]\n",
      " ...\n",
      " [-0.1515113  -0.1515113  -0.1515113  ... -0.11348823 -0.11348823\n",
      "  -0.11348823]\n",
      " [-0.14378984 -0.14378984 -0.14378984 ... -0.10157175 -0.10157175\n",
      "  -0.10157175]\n",
      " [-0.14632946 -0.14632946 -0.14632946 ... -0.13602053 -0.13602053\n",
      "  -0.13602053]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 37 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15234111 -0.15234111 -0.15234111 ... -0.08265202 -0.08265202\n",
      "  -0.08265202]\n",
      " [-0.11639113 -0.11639113 -0.11639113 ... -0.16643983 -0.16643983\n",
      "  -0.16643983]\n",
      " [-0.13167803 -0.13167803 -0.13167803 ... -0.19495237 -0.19495237\n",
      "  -0.19495237]\n",
      " ...\n",
      " [-0.12885728 -0.12885728 -0.12885728 ... -0.15672827 -0.15672827\n",
      "  -0.15672827]\n",
      " [-0.14664137 -0.14664137 -0.14664137 ... -0.16731727 -0.16731727\n",
      "  -0.16731727]\n",
      " [-0.13204515 -0.13204515 -0.13204515 ... -0.13066708 -0.13066708\n",
      "  -0.13066708]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 38 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.10463833 -0.10463833 -0.10463833 ... -0.16859746 -0.16859746\n",
      "  -0.16859746]\n",
      " [-0.18327324 -0.18327324 -0.18327324 ... -0.15318826 -0.15318826\n",
      "  -0.15318826]\n",
      " [-0.13773768 -0.13773768 -0.13773768 ... -0.1530818  -0.1530818\n",
      "  -0.1530818 ]\n",
      " ...\n",
      " [-0.18455307 -0.18455307 -0.18455307 ... -0.11253978 -0.11253978\n",
      "  -0.11253978]\n",
      " [-0.14872415 -0.14872415 -0.14872415 ... -0.17704423 -0.17704423\n",
      "  -0.17704423]\n",
      " [-0.14881009 -0.14881009 -0.14881009 ... -0.15078045 -0.15078045\n",
      "  -0.15078045]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 39 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.17008153 -0.17008153 -0.17008153 ... -0.18683137 -0.18683137\n",
      "  -0.18683137]\n",
      " [-0.16702008 -0.16702008 -0.16702008 ... -0.1734929  -0.1734929\n",
      "  -0.1734929 ]\n",
      " [-0.14861295 -0.14861295 -0.14861295 ... -0.21928674 -0.21928674\n",
      "  -0.21928674]\n",
      " ...\n",
      " [-0.17055207 -0.17055207 -0.17055207 ... -0.17247464 -0.17247464\n",
      "  -0.17247464]\n",
      " [-0.15308519 -0.15308519 -0.15308519 ... -0.14759655 -0.14759655\n",
      "  -0.14759655]\n",
      " [-0.18583004 -0.18583004 -0.18583004 ... -0.20525362 -0.20525362\n",
      "  -0.20525362]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 40 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16127563 -0.16127563 -0.16127563 ... -0.18305017 -0.18305017\n",
      "  -0.18305017]\n",
      " [-0.19229272 -0.19229272 -0.19229272 ... -0.14633477 -0.14633477\n",
      "  -0.14633477]\n",
      " [-0.19745174 -0.19745174 -0.19745174 ... -0.15941185 -0.15941185\n",
      "  -0.15941185]\n",
      " ...\n",
      " [-0.18763411 -0.18763411 -0.18763411 ... -0.18792686 -0.18792686\n",
      "  -0.18792686]\n",
      " [-0.15618752 -0.15618752 -0.15618752 ... -0.23731919 -0.23731919\n",
      "  -0.23731919]\n",
      " [-0.20557746 -0.20557746 -0.20557746 ... -0.1811578  -0.1811578\n",
      "  -0.1811578 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 41 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16767243 -0.16767243 -0.16767243 ... -0.1746719  -0.1746719\n",
      "  -0.1746719 ]\n",
      " [-0.19553    -0.19553    -0.19553    ... -0.17732385 -0.17732385\n",
      "  -0.17732385]\n",
      " [-0.2599875  -0.2599875  -0.2599875  ... -0.1686933  -0.1686933\n",
      "  -0.1686933 ]\n",
      " ...\n",
      " [-0.2059074  -0.2059074  -0.2059074  ... -0.20094897 -0.20094897\n",
      "  -0.20094897]\n",
      " [-0.18584926 -0.18584926 -0.18584926 ... -0.17014623 -0.17014623\n",
      "  -0.17014623]\n",
      " [-0.23590223 -0.23590223 -0.23590223 ... -0.16030055 -0.16030055\n",
      "  -0.16030055]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 42 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20266604 -0.20266604 -0.20266604 ... -0.1730922  -0.1730922\n",
      "  -0.1730922 ]\n",
      " [-0.16007668 -0.16007668 -0.16007668 ... -0.20875275 -0.20875275\n",
      "  -0.20875275]\n",
      " [-0.2233911  -0.2233911  -0.2233911  ... -0.19393447 -0.19393447\n",
      "  -0.19393447]\n",
      " ...\n",
      " [-0.20041469 -0.20041469 -0.20041469 ... -0.161589   -0.161589\n",
      "  -0.161589  ]\n",
      " [-0.21321146 -0.21321146 -0.21321146 ... -0.17679264 -0.17679264\n",
      "  -0.17679264]\n",
      " [-0.16867596 -0.16867596 -0.16867596 ... -0.19762306 -0.19762306\n",
      "  -0.19762306]], shape=(65536, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 43 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.26315174 -0.26315174 -0.26315174 ... -0.20632924 -0.20632924\n",
      "  -0.20632924]\n",
      " [-0.19329208 -0.19329208 -0.19329208 ... -0.20625876 -0.20625876\n",
      "  -0.20625876]\n",
      " [-0.23012948 -0.23012948 -0.23012948 ... -0.24424371 -0.24424371\n",
      "  -0.24424371]\n",
      " ...\n",
      " [-0.2262593  -0.2262593  -0.2262593  ... -0.22234231 -0.22234231\n",
      "  -0.22234231]\n",
      " [-0.20353846 -0.20353846 -0.20353846 ... -0.20778775 -0.20778775\n",
      "  -0.20778775]\n",
      " [-0.19702505 -0.19702505 -0.19702505 ... -0.21742004 -0.21742004\n",
      "  -0.21742004]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 44 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20941475 -0.20941475 -0.20941475 ... -0.22879735 -0.22879735\n",
      "  -0.22879735]\n",
      " [-0.21896069 -0.21896069 -0.21896069 ... -0.23172814 -0.23172814\n",
      "  -0.23172814]\n",
      " [-0.21667118 -0.21667118 -0.21667118 ... -0.21302357 -0.21302357\n",
      "  -0.21302357]\n",
      " ...\n",
      " [-0.20077366 -0.20077366 -0.20077366 ... -0.24974722 -0.24974722\n",
      "  -0.24974722]\n",
      " [-0.22919126 -0.22919126 -0.22919126 ... -0.2074741  -0.2074741\n",
      "  -0.2074741 ]\n",
      " [-0.20116545 -0.20116545 -0.20116545 ... -0.22309692 -0.22309692\n",
      "  -0.22309692]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 45 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.21065114 -0.21065114 -0.21065114 ... -0.24254598 -0.24254598\n",
      "  -0.24254598]\n",
      " [-0.26317322 -0.26317322 -0.26317322 ... -0.24290758 -0.24290758\n",
      "  -0.24290758]\n",
      " [-0.22489735 -0.22489735 -0.22489735 ... -0.22809574 -0.22809574\n",
      "  -0.22809574]\n",
      " ...\n",
      " [-0.24730451 -0.24730451 -0.24730451 ... -0.20688392 -0.20688392\n",
      "  -0.20688392]\n",
      " [-0.25170848 -0.25170848 -0.25170848 ... -0.21429774 -0.21429774\n",
      "  -0.21429774]\n",
      " [-0.20404266 -0.20404266 -0.20404266 ... -0.23616028 -0.23616028\n",
      "  -0.23616028]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 46 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23491627 -0.23491627 -0.23491627 ... -0.24627995 -0.24627995\n",
      "  -0.24627995]\n",
      " [-0.23250955 -0.23250955 -0.23250955 ... -0.21174785 -0.21174785\n",
      "  -0.21174785]\n",
      " [-0.20786193 -0.20786193 -0.20786193 ... -0.26196307 -0.26196307\n",
      "  -0.26196307]\n",
      " ...\n",
      " [-0.26688656 -0.26688656 -0.26688656 ... -0.22311981 -0.22311981\n",
      "  -0.22311981]\n",
      " [-0.24777335 -0.24777335 -0.24777335 ... -0.2504615  -0.2504615\n",
      "  -0.2504615 ]\n",
      " [-0.24965516 -0.24965516 -0.24965516 ... -0.24933492 -0.24933492\n",
      "  -0.24933492]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 47 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23452114 -0.23452114 -0.23452114 ... -0.21214412 -0.21214412\n",
      "  -0.21214412]\n",
      " [-0.25325948 -0.25325948 -0.25325948 ... -0.21796161 -0.21796161\n",
      "  -0.21796161]\n",
      " [-0.2533344  -0.2533344  -0.2533344  ... -0.22835205 -0.22835205\n",
      "  -0.22835205]\n",
      " ...\n",
      " [-0.20976725 -0.20976725 -0.20976725 ... -0.26134884 -0.26134884\n",
      "  -0.26134884]\n",
      " [-0.22285774 -0.22285774 -0.22285774 ... -0.26490912 -0.26490912\n",
      "  -0.26490912]\n",
      " [-0.25237316 -0.25237316 -0.25237316 ... -0.23408599 -0.23408599\n",
      "  -0.23408599]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 48 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.24711835 -0.24711835 -0.24711835 ... -0.28307974 -0.28307974\n",
      "  -0.28307974]\n",
      " [-0.24798271 -0.24798271 -0.24798271 ... -0.2596153  -0.2596153\n",
      "  -0.2596153 ]\n",
      " [-0.23282395 -0.23282395 -0.23282395 ... -0.24117707 -0.24117707\n",
      "  -0.24117707]\n",
      " ...\n",
      " [-0.23466814 -0.23466814 -0.23466814 ... -0.21984717 -0.21984717\n",
      "  -0.21984717]\n",
      " [-0.23673314 -0.23673314 -0.23673314 ... -0.22425437 -0.22425437\n",
      "  -0.22425437]\n",
      " [-0.2085725  -0.2085725  -0.2085725  ... -0.26559567 -0.26559567\n",
      "  -0.26559567]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 49 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20031108 -0.20031108 -0.20031108 ... -0.24550357 -0.24550357\n",
      "  -0.24550357]\n",
      " [-0.22935307 -0.22935307 -0.22935307 ... -0.28199446 -0.28199446\n",
      "  -0.28199446]\n",
      " [-0.25484872 -0.25484872 -0.25484872 ... -0.24498755 -0.24498755\n",
      "  -0.24498755]\n",
      " ...\n",
      " [-0.31743282 -0.31743282 -0.31743282 ... -0.25191927 -0.25191927\n",
      "  -0.25191927]\n",
      " [-0.16408554 -0.16408554 -0.16408554 ... -0.23854835 -0.23854835\n",
      "  -0.23854835]\n",
      " [-0.25009653 -0.25009653 -0.25009653 ... -0.24297011 -0.24297011\n",
      "  -0.24297011]], shape=(65536, 40), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "# train TensorFlow Demo Model, this command will print each iteration's embedding vector\n",
    "tf_results = test_tf_demo(args, init_tensors, *random_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1105e871",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2', '/job:localhost/replica:0/task:0/device:GPU:3', '/job:localhost/replica:0/task:0/device:GPU:4', '/job:localhost/replica:0/task:0/device:GPU:5', '/job:localhost/replica:0/task:0/device:GPU:6', '/job:localhost/replica:0/task:0/device:GPU:7')\n",
      "You are using the plugin with MirroredStrategy.\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:99] Mapping from local_replica_id to device_id:\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 0 -> 0\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 1 -> 1\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 2 -> 2\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 3 -> 3\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 4 -> 4\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 5 -> 5\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 6 -> 6\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 7 -> 7\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:77] Global seed is 153743662\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:78] Local GPU Count: 8\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:79] Global GPU Count: 8\n",
      "2021-12-07 04:48:48.852528: I 2021-12-07 04:48:48.sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119852528] : I Global Replica Id: 1; Local Replica Id: 1\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 0; Local Replica Id: 0\n",
      "2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 2; Local Replica Id: 22021-\n",
      "12-sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] 7 04:48:48.Global Replica Id: 6; Local Replica Id: 6\n",
      "852528: I 2021-12-07 04:48:48.852528: I 2021-12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 5; Local Replica Id: 52021-\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 7; Local Replica Id: 7\n",
      "12-07 04:48:48.852528: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 3; Local Replica Id: 3\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 4; Local Replica Id: 4\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:189] Not all peer to peer access enabled.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_1/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_2/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_3/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_4/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_5/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_6/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_7/\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc_impl/embedding/lookuper/distributed.cc:56] max_vocabulary_size_in_total = 8192\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 5 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 2 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 6 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 1 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 3 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 7 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 0 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 7 initialization done.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 5 initialization done.\n",
      "2021-12-07 04:48:56.852536: I 2021-12-07 04:48:56.852536: I 2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 6 initialization done.\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 0 initialization done.\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 4 start initialization\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 1 initialization done.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 3 initialization done.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 2 initialization done.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 4 initialization done.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/facade.cc:260] SparseOperationKit allocated internal memory.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:213] Loading embedding values to Variable: EmbeddingVariable...\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:307] Allocated temporary buffer for loading embedding values.\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:299] num_total_keys = 8192, while total_max_vocabulary_size = 8192\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:349] Worker 0: Start uploading parameters. Total loop_num = 8\n",
      "2021-12-07 04:48:56.852536: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:223] Loaded embedding values to Variable: EmbeddingVariable.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  0 ------------------------------\n",
      "INFO:tensorflow:batch_all_reduce: 2 all-reduces with algorithm = nccl, num_packs = 1\n",
      "INFO:tensorflow:batch_all_reduce: 2 all-reduces with algorithm = nccl, num_packs = 1\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  1 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.9005419  0.9005419  0.9005419  ... 0.9004803  0.9004803  0.9004803 ]\n",
      " [0.90048057 0.90048057 0.90048057 ... 0.9006749  0.9006749  0.9006749 ]\n",
      " [0.90046704 0.90046704 0.90046704 ... 0.9004606  0.9004606  0.9004606 ]\n",
      " ...\n",
      " [0.90055585 0.90055585 0.90055585 ... 0.9006419  0.9006419  0.9006419 ]\n",
      " [0.90048337 0.90048337 0.90048337 ... 0.90055823 0.90055823 0.90055823]\n",
      " [0.9005347  0.9005347  0.9005347  ... 0.900476   0.900476   0.900476  ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.90049976 0.90049976 0.90049976 ... 0.90054005 0.90054005 0.90054005]\n",
      " [0.9005377  0.9005377  0.9005377  ... 0.900561   0.900561   0.900561  ]\n",
      " [0.90046096 0.90046096 0.90046096 ... 0.9005061  0.9005061  0.9005061 ]\n",
      " ...\n",
      " [0.9005108  0.9005108  0.9005108  ... 0.90058774 0.90058774 0.90058774]\n",
      " [0.90056974 0.90056974 0.90056974 ... 0.9005353  0.9005353  0.9005353 ]\n",
      " [0.9004703  0.9004703  0.9004703  ... 0.90055096 0.90055096 0.90055096]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.9004843  0.9004843  0.9004843  ... 0.90062565 0.90062565 0.90062565]\n",
      " [0.9004948  0.9004948  0.9004948  ... 0.9005251  0.9005251  0.9005251 ]\n",
      " [0.90057486 0.90057486 0.90057486 ... 0.9004644  0.9004644  0.9004644 ]\n",
      " ...\n",
      " [0.9005201  0.9005201  0.9005201  ... 0.9006022  0.9006022  0.9006022 ]\n",
      " [0.90053344 0.90053344 0.90053344 ... 0.9005527  0.9005527  0.9005527 ]\n",
      " [0.90050566 0.90050566 0.90050566 ... 0.90054536 0.90054536 0.90054536]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.9005505  0.9005505  0.9005505  ... 0.9004445  0.9004445  0.9004445 ]\n",
      " [0.9004284  0.9004284  0.9004284  ... 0.90058553 0.90058553 0.90058553]\n",
      " [0.900484   0.900484   0.900484   ... 0.9005472  0.9005472  0.9005472 ]\n",
      " ...\n",
      " [0.90060425 0.90060425 0.90060425 ... 0.90060294 0.90060294 0.90060294]\n",
      " [0.9004948  0.9004948  0.9004948  ... 0.9005274  0.9005274  0.9005274 ]\n",
      " [0.90051395 0.90051395 0.90051395 ... 0.90045595 0.90045595 0.90045595]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.9004965  0.9004965  0.9004965  ... 0.90053976 0.90053976 0.90053976]\n",
      " [0.9004981  0.9004981  0.9004981  ... 0.90055823 0.90055823 0.90055823]\n",
      " [0.9005179  0.9005179  0.9005179  ... 0.9004543  0.9004543  0.9004543 ]\n",
      " ...\n",
      " [0.90054846 0.90054846 0.90054846 ... 0.90057015 0.90057015 0.90057015]\n",
      " [0.90046716 0.90046716 0.90046716 ... 0.90046054 0.90046054 0.90046054]\n",
      " [0.90053934 0.90053934 0.90053934 ... 0.90054965 0.90054965 0.90054965]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.90046614 0.90046614 0.90046614 ... 0.900621   0.900621   0.900621  ]\n",
      " [0.9005136  0.9005136  0.9005136  ... 0.9005379  0.9005379  0.9005379 ]\n",
      " [0.9005165  0.9005165  0.9005165  ... 0.90058345 0.90058345 0.90058345]\n",
      " ...\n",
      " [0.90047544 0.90047544 0.90047544 ... 0.9005462  0.9005462  0.9005462 ]\n",
      " [0.9005834  0.9005834  0.9005834  ... 0.90074575 0.90074575 0.90074575]\n",
      " [0.90053225 0.90053225 0.90053225 ... 0.9005634  0.9005634  0.9005634 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.9005109  0.9005109  0.9005109  ... 0.90050507 0.90050507 0.90050507]\n",
      " [0.90045345 0.90045345 0.90045345 ... 0.90060353 0.90060353 0.90060353]\n",
      " [0.900522   0.900522   0.900522   ... 0.9005786  0.9005786  0.9005786 ]\n",
      " ...\n",
      " [0.90047705 0.90047705 0.90047705 ... 0.90050215 0.90050215 0.90050215]\n",
      " [0.90047324 0.90047324 0.90047324 ... 0.9004742  0.9004742  0.9004742 ]\n",
      " [0.9004836  0.9004836  0.9004836  ... 0.9005233  0.9005233  0.9005233 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.90058404 0.90058404 0.90058404 ... 0.90049005 0.90049005 0.90049005]\n",
      " [0.90061486 0.90061486 0.90061486 ... 0.90051323 0.90051323 0.90051323]\n",
      " [0.90061486 0.90061486 0.90061486 ... 0.9005427  0.9005427  0.9005427 ]\n",
      " ...\n",
      " [0.9005053  0.9005053  0.9005053  ... 0.90053415 0.90053415 0.90053415]\n",
      " [0.900587   0.900587   0.900587   ... 0.9005064  0.9005064  0.9005064 ]\n",
      " [0.9004429  0.9004429  0.9004429  ... 0.9004999  0.9004999  0.9004999 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  2 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.8012225  0.8012225  0.8012225  ... 0.80078465 0.80078465 0.80078465]\n",
      " [0.8010973  0.8010973  0.8010973  ... 0.8017616  0.8017616  0.8017616 ]\n",
      " [0.801127   0.801127   0.801127   ... 0.8008523  0.8008523  0.8008523 ]\n",
      " ...\n",
      " [0.8008735  0.8008735  0.8008735  ... 0.80163956 0.80163956 0.80163956]\n",
      " [0.8012477  0.8012477  0.8012477  ... 0.80182624 0.80182624 0.80182624]\n",
      " [0.8016155  0.8016155  0.8016155  ... 0.8009877  0.8009877  0.8009877 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.80176467 0.80176467 0.80176467 ... 0.8020307  0.8020307  0.8020307 ]\n",
      " [0.8012211  0.8012211  0.8012211  ... 0.8014598  0.8014598  0.8014598 ]\n",
      " [0.8026409  0.8026409  0.8026409  ... 0.8024336  0.8024336  0.8024336 ]\n",
      " ...\n",
      " [0.80134475 0.80134475 0.80134475 ... 0.8021525  0.8021525  0.8021525 ]\n",
      " [0.8059662  0.8059662  0.8059662  ... 0.80127954 0.80127954 0.80127954]\n",
      " [0.80102426 0.80102426 0.80102426 ... 0.80076885 0.80076885 0.80076885]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.80111074 0.80111074 0.80111074 ... 0.80213773 0.80213773 0.80213773]\n",
      " [0.80295086 0.80295086 0.80295086 ... 0.8010617  0.8010617  0.8010617 ]\n",
      " [0.80215347 0.80215347 0.80215347 ... 0.80114686 0.80114686 0.80114686]\n",
      " ...\n",
      " [0.8009839  0.8009839  0.8009839  ... 0.80149263 0.80149263 0.80149263]\n",
      " [0.8017583  0.8017583  0.8017583  ... 0.80121994 0.80121994 0.80121994]\n",
      " [0.80212677 0.80212677 0.80212677 ... 0.80101764 0.80101764 0.80101764]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.8011631  0.8011631  0.8011631  ... 0.8010175  0.8010175  0.8010175 ]\n",
      " [0.8019387  0.8019387  0.8019387  ... 0.8015175  0.8015175  0.8015175 ]\n",
      " [0.8015401  0.8015401  0.8015401  ... 0.80142486 0.80142486 0.80142486]\n",
      " ...\n",
      " [0.800875   0.800875   0.800875   ... 0.801806   0.801806   0.801806  ]\n",
      " [0.8025913  0.8025913  0.8025913  ... 0.80273163 0.80273163 0.80273163]\n",
      " [0.8010912  0.8010912  0.8010912  ... 0.8009335  0.8009335  0.8009335 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.8014086  0.8014086  0.8014086  ... 0.8013951  0.8013951  0.8013951 ]\n",
      " [0.80394804 0.80394804 0.80394804 ... 0.8015265  0.8015265  0.8015265 ]\n",
      " [0.80270183 0.80270183 0.80270183 ... 0.8019798  0.8019798  0.8019798 ]\n",
      " ...\n",
      " [0.8022102  0.8022102  0.8022102  ... 0.8016084  0.8016084  0.8016084 ]\n",
      " [0.8008057  0.8008057  0.8008057  ... 0.8008648  0.8008648  0.8008648 ]\n",
      " [0.80131793 0.80131793 0.80131793 ... 0.8009633  0.8009633  0.8009633 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.80115503 0.80115503 0.80115503 ... 0.80244094 0.80244094 0.80244094]\n",
      " [0.80221444 0.80221444 0.80221444 ... 0.80273414 0.80273414 0.80273414]\n",
      " [0.801698   0.801698   0.801698   ... 0.80097187 0.80097187 0.80097187]\n",
      " ...\n",
      " [0.8013543  0.8013543  0.8013543  ... 0.8018955  0.8018955  0.8018955 ]\n",
      " [0.80240816 0.80240816 0.80240816 ... 0.8018257  0.8018257  0.8018257 ]\n",
      " [0.8011422  0.8011422  0.8011422  ... 0.8029331  0.8029331  0.8029331 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.80213886 0.80213886 0.80213886 ... 0.8017629  0.8017629  0.8017629 ]\n",
      " [0.8017787  0.8017787  0.8017787  ... 0.80201805 0.80201805 0.80201805]\n",
      " [0.8020815  0.8020815  0.8020815  ... 0.80325174 0.80325174 0.80325174]\n",
      " ...\n",
      " [0.80081236 0.80081236 0.80081236 ... 0.8009473  0.8009473  0.8009473 ]\n",
      " [0.801543   0.801543   0.801543   ... 0.8019837  0.8019837  0.8019837 ]\n",
      " [0.8021473  0.8021473  0.8021473  ... 0.8019671  0.8019671  0.8019671 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.8015705  0.8015705  0.8015705  ... 0.8016241  0.8016241  0.8016241 ]\n",
      " [0.80122256 0.80122256 0.80122256 ... 0.8015234  0.8015234  0.8015234 ]\n",
      " [0.8019392  0.8019392  0.8019392  ... 0.8015372  0.8015372  0.8015372 ]\n",
      " ...\n",
      " [0.80131733 0.80131733 0.80131733 ... 0.8011143  0.8011143  0.8011143 ]\n",
      " [0.80127215 0.80127215 0.80127215 ... 0.801376   0.801376   0.801376  ]\n",
      " [0.8010522  0.8010522  0.8010522  ... 0.80395603 0.80395603 0.80395603]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  3 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.70422626 0.70422626 0.70422626 ... 0.7052028  0.7052028  0.7052028 ]\n",
      " [0.7036482  0.7036482  0.7036482  ... 0.7032463  0.7032463  0.7032463 ]\n",
      " [0.701295   0.701295   0.701295   ... 0.7022344  0.7022344  0.7022344 ]\n",
      " ...\n",
      " [0.7025738  0.7025738  0.7025738  ... 0.7037034  0.7037034  0.7037034 ]\n",
      " [0.7022596  0.7022596  0.7022596  ... 0.7078545  0.7078545  0.7078545 ]\n",
      " [0.70445216 0.70445216 0.70445216 ... 0.704602   0.704602   0.704602  ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.7023215  0.7023215  0.7023215  ... 0.70324075 0.70324075 0.70324075]\n",
      " [0.702116   0.702116   0.702116   ... 0.7016717  0.7016717  0.7016717 ]\n",
      " [0.7035055  0.7035055  0.7035055  ... 0.7026157  0.7026157  0.7026157 ]\n",
      " ...\n",
      " [0.7045913  0.7045913  0.7045913  ... 0.70173776 0.70173776 0.70173776]\n",
      " [0.702466   0.702466   0.702466   ... 0.7023295  0.7023295  0.7023295 ]\n",
      " [0.7024561  0.7024561  0.7024561  ... 0.7030035  0.7030035  0.7030035 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.70479333 0.70479333 0.70479333 ... 0.70407945 0.70407945 0.70407945]\n",
      " [0.7032777  0.7032777  0.7032777  ... 0.7032692  0.7032692  0.7032692 ]\n",
      " [0.7030939  0.7030939  0.7030939  ... 0.705856   0.705856   0.705856  ]\n",
      " ...\n",
      " [0.7032618  0.7032618  0.7032618  ... 0.7036378  0.7036378  0.7036378 ]\n",
      " [0.703318   0.703318   0.703318   ... 0.70335853 0.70335853 0.70335853]\n",
      " [0.7037716  0.7037716  0.7037716  ... 0.70655346 0.70655346 0.70655346]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.7036817  0.7036817  0.7036817  ... 0.7029312  0.7029312  0.7029312 ]\n",
      " [0.7033701  0.7033701  0.7033701  ... 0.70136344 0.70136344 0.70136344]\n",
      " [0.7019788  0.7019788  0.7019788  ... 0.7033566  0.7033566  0.7033566 ]\n",
      " ...\n",
      " [0.70392275 0.70392275 0.70392275 ... 0.70137787 0.70137787 0.70137787]\n",
      " [0.7032561  0.7032561  0.7032561  ... 0.7023928  0.7023928  0.7023928 ]\n",
      " [0.70434594 0.70434594 0.70434594 ... 0.70288485 0.70288485 0.70288485]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.7028553  0.7028553  0.7028553  ... 0.7046065  0.7046065  0.7046065 ]\n",
      " [0.7042153  0.7042153  0.7042153  ... 0.7026787  0.7026787  0.7026787 ]\n",
      " [0.7025721  0.7025721  0.7025721  ... 0.70724994 0.70724994 0.70724994]\n",
      " ...\n",
      " [0.7031083  0.7031083  0.7031083  ... 0.70434356 0.70434356 0.70434356]\n",
      " [0.70130396 0.70130396 0.70130396 ... 0.70846826 0.70846826 0.70846826]\n",
      " [0.70290726 0.70290726 0.70290726 ... 0.7031524  0.7031524  0.7031524 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.70230883 0.70230883 0.70230883 ... 0.7054386  0.7054386  0.7054386 ]\n",
      " [0.70344096 0.70344096 0.70344096 ... 0.70307755 0.70307755 0.70307755]\n",
      " [0.7032068  0.7032068  0.7032068  ... 0.7042556  0.7042556  0.7042556 ]\n",
      " ...\n",
      " [0.7036709  0.7036709  0.7036709  ... 0.70273495 0.70273495 0.70273495]\n",
      " [0.7057643  0.7057643  0.7057643  ... 0.705743   0.705743   0.705743  ]\n",
      " [0.70433813 0.70433813 0.70433813 ... 0.70365953 0.70365953 0.70365953]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.7040535  0.7040535  0.7040535  ... 0.7036978  0.7036978  0.7036978 ]\n",
      " [0.7048917  0.7048917  0.7048917  ... 0.70409596 0.70409596 0.70409596]\n",
      " [0.7040311  0.7040311  0.7040311  ... 0.7045357  0.7045357  0.7045357 ]\n",
      " ...\n",
      " [0.7027895  0.7027895  0.7027895  ... 0.7022587  0.7022587  0.7022587 ]\n",
      " [0.7019428  0.7019428  0.7019428  ... 0.7058565  0.7058565  0.7058565 ]\n",
      " [0.70411134 0.70411134 0.70411134 ... 0.7036616  0.7036616  0.7036616 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.7028538  0.7028538  0.7028538  ... 0.70207703 0.70207703 0.70207703]\n",
      " [0.7048682  0.7048682  0.7048682  ... 0.70207304 0.70207304 0.70207304]\n",
      " [0.70413727 0.70413727 0.70413727 ... 0.7037903  0.7037903  0.7037903 ]\n",
      " ...\n",
      " [0.7018771  0.7018771  0.7018771  ... 0.706771   0.706771   0.706771  ]\n",
      " [0.7038691  0.7038691  0.7038691  ... 0.70275795 0.70275795 0.70275795]\n",
      " [0.7041686  0.7041686  0.7041686  ... 0.70362675 0.70362675 0.70362675]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  4 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.60574055 0.60574055 0.60574055 ... 0.6049524  0.6049524  0.6049524 ]\n",
      " [0.6066351  0.6066351  0.6066351  ... 0.6060442  0.6060442  0.6060442 ]\n",
      " [0.6072282  0.6072282  0.6072282  ... 0.6045385  0.6045385  0.6045385 ]\n",
      " ...\n",
      " [0.6054753  0.6054753  0.6054753  ... 0.6040589  0.6040589  0.6040589 ]\n",
      " [0.60539    0.60539    0.60539    ... 0.6054742  0.6054742  0.6054742 ]\n",
      " [0.6068313  0.6068313  0.6068313  ... 0.60812074 0.60812074 0.60812074]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.6078492  0.6078492  0.6078492  ... 0.6111247  0.6111247  0.6111247 ]\n",
      " [0.6049961  0.6049961  0.6049961  ... 0.60674167 0.60674167 0.60674167]\n",
      " [0.6046079  0.6046079  0.6046079  ... 0.6050034  0.6050034  0.6050034 ]\n",
      " ...\n",
      " [0.6083204  0.6083204  0.6083204  ... 0.6127972  0.6127972  0.6127972 ]\n",
      " [0.6149222  0.6149222  0.6149222  ... 0.60844254 0.60844254 0.60844254]\n",
      " [0.6091602  0.6091602  0.6091602  ... 0.6053159  0.6053159  0.6053159 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.6075534  0.6075534  0.6075534  ... 0.60568714 0.60568714 0.60568714]\n",
      " [0.6057408  0.6057408  0.6057408  ... 0.604336   0.604336   0.604336  ]\n",
      " [0.6149405  0.6149405  0.6149405  ... 0.6105539  0.6105539  0.6105539 ]\n",
      " ...\n",
      " [0.60493624 0.60493624 0.60493624 ... 0.6050907  0.6050907  0.6050907 ]\n",
      " [0.6054564  0.6054564  0.6054564  ... 0.6073811  0.6073811  0.6073811 ]\n",
      " [0.60686314 0.60686314 0.60686314 ... 0.6038236  0.6038236  0.6038236 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.60847324 0.60847324 0.60847324 ... 0.60567933 0.60567933 0.60567933]\n",
      " [0.60806715 0.60806715 0.60806715 ... 0.60570616 0.60570616 0.60570616]\n",
      " [0.6066627  0.6066627  0.6066627  ... 0.6075407  0.6075407  0.6075407 ]\n",
      " ...\n",
      " [0.6067428  0.6067428  0.6067428  ... 0.61036146 0.61036146 0.61036146]\n",
      " [0.6066314  0.6066314  0.6066314  ... 0.6030339  0.6030339  0.6030339 ]\n",
      " [0.6070837  0.6070837  0.6070837  ... 0.6067776  0.6067776  0.6067776 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.60719544 0.60719544 0.60719544 ... 0.6075876  0.6075876  0.6075876 ]\n",
      " [0.6042855  0.6042855  0.6042855  ... 0.6058589  0.6058589  0.6058589 ]\n",
      " [0.6051324  0.6051324  0.6051324  ... 0.60511726 0.60511726 0.60511726]\n",
      " ...\n",
      " [0.60927033 0.60927033 0.60927033 ... 0.6068082  0.6068082  0.6068082 ]\n",
      " [0.6114415  0.6114415  0.6114415  ... 0.6052427  0.6052427  0.6052427 ]\n",
      " [0.6080282  0.6080282  0.6080282  ... 0.6082679  0.6082679  0.6082679 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.61047125 0.61047125 0.61047125 ... 0.6037925  0.6037925  0.6037925 ]\n",
      " [0.6066753  0.6066753  0.6066753  ... 0.6045574  0.6045574  0.6045574 ]\n",
      " [0.60562366 0.60562366 0.60562366 ... 0.60574853 0.60574853 0.60574853]\n",
      " ...\n",
      " [0.60897475 0.60897475 0.60897475 ... 0.6079279  0.6079279  0.6079279 ]\n",
      " [0.61697644 0.61697644 0.61697644 ... 0.6059196  0.6059196  0.6059196 ]\n",
      " [0.6043109  0.6043109  0.6043109  ... 0.608747   0.608747   0.608747  ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.60462177 0.60462177 0.60462177 ... 0.6061893  0.6061893  0.6061893 ]\n",
      " [0.60577273 0.60577273 0.60577273 ... 0.60894907 0.60894907 0.60894907]\n",
      " [0.6087152  0.6087152  0.6087152  ... 0.61593413 0.61593413 0.61593413]\n",
      " ...\n",
      " [0.6078811  0.6078811  0.6078811  ... 0.60969913 0.60969913 0.60969913]\n",
      " [0.60535204 0.60535204 0.60535204 ... 0.6064161  0.6064161  0.6064161 ]\n",
      " [0.60716414 0.60716414 0.60716414 ... 0.6064824  0.6064824  0.6064824 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.60425115 0.60425115 0.60425115 ... 0.6035463  0.6035463  0.6035463 ]\n",
      " [0.6053508  0.6053508  0.6053508  ... 0.6022028  0.6022028  0.6022028 ]\n",
      " [0.60736257 0.60736257 0.60736257 ... 0.6081464  0.6081464  0.6081464 ]\n",
      " ...\n",
      " [0.60645854 0.60645854 0.60645854 ... 0.60563594 0.60563594 0.60563594]\n",
      " [0.60451645 0.60451645 0.60451645 ... 0.6048237  0.6048237  0.6048237 ]\n",
      " [0.6046561  0.6046561  0.6046561  ... 0.6037593  0.6037593  0.6037593 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  5 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.51534927 0.51534927 0.51534927 ... 0.51107526 0.51107526 0.51107526]\n",
      " [0.5077996  0.5077996  0.5077996  ... 0.5091748  0.5091748  0.5091748 ]\n",
      " [0.5132122  0.5132122  0.5132122  ... 0.5186943  0.5186943  0.5186943 ]\n",
      " ...\n",
      " [0.5080771  0.5080771  0.5080771  ... 0.51086533 0.51086533 0.51086533]\n",
      " [0.5110096  0.5110096  0.5110096  ... 0.5139279  0.5139279  0.5139279 ]\n",
      " [0.51000637 0.51000637 0.51000637 ... 0.51075566 0.51075566 0.51075566]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.5117598  0.5117598  0.5117598  ... 0.5180596  0.5180596  0.5180596 ]\n",
      " [0.5150682  0.5150682  0.5150682  ... 0.5090677  0.5090677  0.5090677 ]\n",
      " [0.51345956 0.51345956 0.51345956 ... 0.5124791  0.5124791  0.5124791 ]\n",
      " ...\n",
      " [0.51130927 0.51130927 0.51130927 ... 0.51552176 0.51552176 0.51552176]\n",
      " [0.5106223  0.5106223  0.5106223  ... 0.5117562  0.5117562  0.5117562 ]\n",
      " [0.5127562  0.5127562  0.5127562  ... 0.5133672  0.5133672  0.5133672 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.5121434  0.5121434  0.5121434  ... 0.5077224  0.5077224  0.5077224 ]\n",
      " [0.51223534 0.51223534 0.51223534 ... 0.50904137 0.50904137 0.50904137]\n",
      " [0.5095789  0.5095789  0.5095789  ... 0.50869244 0.50869244 0.50869244]\n",
      " ...\n",
      " [0.50792134 0.50792134 0.50792134 ... 0.5128337  0.5128337  0.5128337 ]\n",
      " [0.5127207  0.5127207  0.5127207  ... 0.5135564  0.5135564  0.5135564 ]\n",
      " [0.5110458  0.5110458  0.5110458  ... 0.5147052  0.5147052  0.5147052 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.5103754  0.5103754  0.5103754  ... 0.51761323 0.51761323 0.51761323]\n",
      " [0.50856125 0.50856125 0.50856125 ... 0.5138728  0.5138728  0.5138728 ]\n",
      " [0.5157521  0.5157521  0.5157521  ... 0.51360106 0.51360106 0.51360106]\n",
      " ...\n",
      " [0.51375943 0.51375943 0.51375943 ... 0.5134803  0.5134803  0.5134803 ]\n",
      " [0.5152999  0.5152999  0.5152999  ... 0.5084388  0.5084388  0.5084388 ]\n",
      " [0.51385915 0.51385915 0.51385915 ... 0.5137102  0.5137102  0.5137102 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.51322865 0.51322865 0.51322865 ... 0.50758433 0.50758433 0.50758433]\n",
      " [0.51454425 0.51454425 0.51454425 ... 0.5120248  0.5120248  0.5120248 ]\n",
      " [0.5072035  0.5072035  0.5072035  ... 0.5081138  0.5081138  0.5081138 ]\n",
      " ...\n",
      " [0.5096794  0.5096794  0.5096794  ... 0.5115247  0.5115247  0.5115247 ]\n",
      " [0.511788   0.511788   0.511788   ... 0.51247275 0.51247275 0.51247275]\n",
      " [0.5145926  0.5145926  0.5145926  ... 0.5160629  0.5160629  0.5160629 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.50896764 0.50896764 0.50896764 ... 0.51619434 0.51619434 0.51619434]\n",
      " [0.5092937  0.5092937  0.5092937  ... 0.50735414 0.50735414 0.50735414]\n",
      " [0.51277053 0.51277053 0.51277053 ... 0.50896883 0.50896883 0.50896883]\n",
      " ...\n",
      " [0.5145924  0.5145924  0.5145924  ... 0.5124101  0.5124101  0.5124101 ]\n",
      " [0.51271355 0.51271355 0.51271355 ... 0.5154177  0.5154177  0.5154177 ]\n",
      " [0.508103   0.508103   0.508103   ... 0.5138626  0.5138626  0.5138626 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.50537574 0.50537574 0.50537574 ... 0.5116946  0.5116946  0.5116946 ]\n",
      " [0.5100052  0.5100052  0.5100052  ... 0.51539385 0.51539385 0.51539385]\n",
      " [0.5144923  0.5144923  0.5144923  ... 0.51152545 0.51152545 0.51152545]\n",
      " ...\n",
      " [0.5065424  0.5065424  0.5065424  ... 0.5129912  0.5129912  0.5129912 ]\n",
      " [0.5099993  0.5099993  0.5099993  ... 0.50947994 0.50947994 0.50947994]\n",
      " [0.50922656 0.50922656 0.50922656 ... 0.5090768  0.5090768  0.5090768 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.50985205 0.50985205 0.50985205 ... 0.5110366  0.5110366  0.5110366 ]\n",
      " [0.5092987  0.5092987  0.5092987  ... 0.5118404  0.5118404  0.5118404 ]\n",
      " [0.51229787 0.51229787 0.51229787 ... 0.51041836 0.51041836 0.51041836]\n",
      " ...\n",
      " [0.5129001  0.5129001  0.5129001  ... 0.513555   0.513555   0.513555  ]\n",
      " [0.5107131  0.5107131  0.5107131  ... 0.5182584  0.5182584  0.5182584 ]\n",
      " [0.51004165 0.51004165 0.51004165 ... 0.51426864 0.51426864 0.51426864]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  6 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.42153543 0.42153543 0.42153543 ... 0.42519388 0.42519388 0.42519388]\n",
      " [0.4170929  0.4170929  0.4170929  ... 0.41547838 0.41547838 0.41547838]\n",
      " [0.4118647  0.4118647  0.4118647  ... 0.4137937  0.4137937  0.4137937 ]\n",
      " ...\n",
      " [0.41602445 0.41602445 0.41602445 ... 0.4223327  0.4223327  0.4223327 ]\n",
      " [0.41451913 0.41451913 0.41451913 ... 0.4199844  0.4199844  0.4199844 ]\n",
      " [0.41644222 0.41644222 0.41644222 ... 0.41922688 0.41922688 0.41922688]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.4137619  0.4137619  0.4137619  ... 0.41629443 0.41629443 0.41629443]\n",
      " [0.4158877  0.4158877  0.4158877  ... 0.42044166 0.42044166 0.42044166]\n",
      " [0.4148597  0.4148597  0.4148597  ... 0.41802114 0.41802114 0.41802114]\n",
      " ...\n",
      " [0.41918486 0.41918486 0.41918486 ... 0.41373587 0.41373587 0.41373587]\n",
      " [0.41620752 0.41620752 0.41620752 ... 0.420725   0.420725   0.420725  ]\n",
      " [0.42256388 0.42256388 0.42256388 ... 0.41387692 0.41387692 0.41387692]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.4175743  0.4175743  0.4175743  ... 0.42591095 0.42591095 0.42591095]\n",
      " [0.41689837 0.41689837 0.41689837 ... 0.41712895 0.41712895 0.41712895]\n",
      " [0.40878794 0.40878794 0.40878794 ... 0.4210766  0.4210766  0.4210766 ]\n",
      " ...\n",
      " [0.41619045 0.41619045 0.41619045 ... 0.42117926 0.42117926 0.42117926]\n",
      " [0.4165277  0.4165277  0.4165277  ... 0.41435412 0.41435412 0.41435412]\n",
      " [0.4170459  0.4170459  0.4170459  ... 0.41607708 0.41607708 0.41607708]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.4196428  0.4196428  0.4196428  ... 0.4197427  0.4197427  0.4197427 ]\n",
      " [0.41923004 0.41923004 0.41923004 ... 0.41890043 0.41890043 0.41890043]\n",
      " [0.4168408  0.4168408  0.4168408  ... 0.41725105 0.41725105 0.41725105]\n",
      " ...\n",
      " [0.42562124 0.42562124 0.42562124 ... 0.41681376 0.41681376 0.41681376]\n",
      " [0.413693   0.413693   0.413693   ... 0.42089516 0.42089516 0.42089516]\n",
      " [0.4167804  0.4167804  0.4167804  ... 0.412577   0.412577   0.412577  ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.41434556 0.41434556 0.41434556 ... 0.41782805 0.41782805 0.41782805]\n",
      " [0.41765755 0.41765755 0.41765755 ... 0.4121918  0.4121918  0.4121918 ]\n",
      " [0.41853532 0.41853532 0.41853532 ... 0.41287488 0.41287488 0.41287488]\n",
      " ...\n",
      " [0.42845628 0.42845628 0.42845628 ... 0.41488355 0.41488355 0.41488355]\n",
      " [0.41249967 0.41249967 0.41249967 ... 0.41657358 0.41657358 0.41657358]\n",
      " [0.4212286  0.4212286  0.4212286  ... 0.42126328 0.42126328 0.42126328]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.41734767 0.41734767 0.41734767 ... 0.43086398 0.43086398 0.43086398]\n",
      " [0.41954458 0.41954458 0.41954458 ... 0.4265362  0.4265362  0.4265362 ]\n",
      " [0.4158059  0.4158059  0.4158059  ... 0.4232058  0.4232058  0.4232058 ]\n",
      " ...\n",
      " [0.4144219  0.4144219  0.4144219  ... 0.4202456  0.4202456  0.4202456 ]\n",
      " [0.41714352 0.41714352 0.41714352 ... 0.41474175 0.41474175 0.41474175]\n",
      " [0.4194298  0.4194298  0.4194298  ... 0.4209739  0.4209739  0.4209739 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.41202855 0.41202855 0.41202855 ... 0.41448775 0.41448775 0.41448775]\n",
      " [0.42272705 0.42272705 0.42272705 ... 0.41926816 0.41926816 0.41926816]\n",
      " [0.4150994  0.4150994  0.4150994  ... 0.42195714 0.42195714 0.42195714]\n",
      " ...\n",
      " [0.42636353 0.42636353 0.42636353 ... 0.41301453 0.41301453 0.41301453]\n",
      " [0.41750285 0.41750285 0.41750285 ... 0.41845286 0.41845286 0.41845286]\n",
      " [0.41332984 0.41332984 0.41332984 ... 0.41409668 0.41409668 0.41409668]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.4179835  0.4179835  0.4179835  ... 0.41236973 0.41236973 0.41236973]\n",
      " [0.42133448 0.42133448 0.42133448 ... 0.4160627  0.4160627  0.4160627 ]\n",
      " [0.41423    0.41423    0.41423    ... 0.4164468  0.4164468  0.4164468 ]\n",
      " ...\n",
      " [0.4167164  0.4167164  0.4167164  ... 0.4097301  0.4097301  0.4097301 ]\n",
      " [0.41558528 0.41558528 0.41558528 ... 0.41867378 0.41867378 0.41867378]\n",
      " [0.42026323 0.42026323 0.42026323 ... 0.41544208 0.41544208 0.41544208]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  7 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.33165017 0.33165017 0.33165017 ... 0.33700365 0.33700365 0.33700365]\n",
      " [0.32833594 0.32833594 0.32833594 ... 0.32462552 0.32462552 0.32462552]\n",
      " [0.32871684 0.32871684 0.32871684 ... 0.32887527 0.32887527 0.32887527]\n",
      " ...\n",
      " [0.32310146 0.32310146 0.32310146 ... 0.33558953 0.33558953 0.33558953]\n",
      " [0.3303491  0.3303491  0.3303491  ... 0.33116546 0.33116546 0.33116546]\n",
      " [0.3204049  0.3204049  0.3204049  ... 0.32539573 0.32539573 0.32539573]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.3397804  0.3397804  0.3397804  ... 0.3230064  0.3230064  0.3230064 ]\n",
      " [0.32889685 0.32889685 0.32889685 ... 0.317527   0.317527   0.317527  ]\n",
      " [0.33834144 0.33834144 0.33834144 ... 0.32701278 0.32701278 0.32701278]\n",
      " ...\n",
      " [0.33635855 0.33635855 0.33635855 ... 0.33034858 0.33034858 0.33034858]\n",
      " [0.32049954 0.32049954 0.32049954 ... 0.3290459  0.3290459  0.3290459 ]\n",
      " [0.33664024 0.33664024 0.33664024 ... 0.32425246 0.32425246 0.32425246]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.32309508 0.32309508 0.32309508 ... 0.33279473 0.33279473 0.33279473]\n",
      " [0.328422   0.328422   0.328422   ... 0.32514685 0.32514685 0.32514685]\n",
      " [0.32916662 0.32916662 0.32916662 ... 0.3318566  0.3318566  0.3318566 ]\n",
      " ...\n",
      " [0.33031064 0.33031064 0.33031064 ... 0.33011085 0.33011085 0.33011085]\n",
      " [0.3299008  0.3299008  0.3299008  ... 0.32333288 0.32333288 0.32333288]\n",
      " [0.32452384 0.32452384 0.32452384 ... 0.3212864  0.3212864  0.3212864 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.33306715 0.33306715 0.33306715 ... 0.3400076  0.3400076  0.3400076 ]\n",
      " [0.3262299  0.3262299  0.3262299  ... 0.32737008 0.32737008 0.32737008]\n",
      " [0.3298506  0.3298506  0.3298506  ... 0.33479002 0.33479002 0.33479002]\n",
      " ...\n",
      " [0.32809243 0.32809243 0.32809243 ... 0.3189497  0.3189497  0.3189497 ]\n",
      " [0.32513154 0.32513154 0.32513154 ... 0.33131972 0.33131972 0.33131972]\n",
      " [0.33130988 0.33130988 0.33130988 ... 0.3317049  0.3317049  0.3317049 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.33337983 0.33337983 0.33337983 ... 0.32134408 0.32134408 0.32134408]\n",
      " [0.32908538 0.32908538 0.32908538 ... 0.33216664 0.33216664 0.33216664]\n",
      " [0.33518845 0.33518845 0.33518845 ... 0.32886347 0.32886347 0.32886347]\n",
      " ...\n",
      " [0.32812142 0.32812142 0.32812142 ... 0.328399   0.328399   0.328399  ]\n",
      " [0.33065918 0.33065918 0.33065918 ... 0.32885915 0.32885915 0.32885915]\n",
      " [0.32276678 0.32276678 0.32276678 ... 0.32906976 0.32906976 0.32906976]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.3280829  0.3280829  0.3280829  ... 0.3284577  0.3284577  0.3284577 ]\n",
      " [0.334884   0.334884   0.334884   ... 0.34313324 0.34313324 0.34313324]\n",
      " [0.3278158  0.3278158  0.3278158  ... 0.3316633  0.3316633  0.3316633 ]\n",
      " ...\n",
      " [0.3304994  0.3304994  0.3304994  ... 0.32254183 0.32254183 0.32254183]\n",
      " [0.33355236 0.33355236 0.33355236 ... 0.33379424 0.33379424 0.33379424]\n",
      " [0.32968652 0.32968652 0.32968652 ... 0.32745725 0.32745725 0.32745725]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.33424404 0.33424404 0.33424404 ... 0.32742712 0.32742712 0.32742712]\n",
      " [0.32693008 0.32693008 0.32693008 ... 0.33252305 0.33252305 0.33252305]\n",
      " [0.33297223 0.33297223 0.33297223 ... 0.33109564 0.33109564 0.33109564]\n",
      " ...\n",
      " [0.31977686 0.31977686 0.31977686 ... 0.32361498 0.32361498 0.32361498]\n",
      " [0.3275146  0.3275146  0.3275146  ... 0.33088252 0.33088252 0.33088252]\n",
      " [0.3217108  0.3217108  0.3217108  ... 0.3309523  0.3309523  0.3309523 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.32837084 0.32837084 0.32837084 ... 0.3417757  0.3417757  0.3417757 ]\n",
      " [0.32294804 0.32294804 0.32294804 ... 0.32462448 0.32462448 0.32462448]\n",
      " [0.32892948 0.32892948 0.32892948 ... 0.32407635 0.32407635 0.32407635]\n",
      " ...\n",
      " [0.32006192 0.32006192 0.32006192 ... 0.33087918 0.33087918 0.33087918]\n",
      " [0.32935303 0.32935303 0.32935303 ... 0.32110265 0.32110265 0.32110265]\n",
      " [0.32427692 0.32427692 0.32427692 ... 0.32734925 0.32734925 0.32734925]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  8 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.2444183  0.2444183  0.2444183  ... 0.24178997 0.24178997 0.24178997]\n",
      " [0.2394118  0.2394118  0.2394118  ... 0.24692377 0.24692377 0.24692377]\n",
      " [0.24404061 0.24404061 0.24404061 ... 0.2439357  0.2439357  0.2439357 ]\n",
      " ...\n",
      " [0.2451255  0.2451255  0.2451255  ... 0.23931435 0.23931435 0.23931435]\n",
      " [0.24269496 0.24269496 0.24269496 ... 0.23764302 0.23764302 0.23764302]\n",
      " [0.23211774 0.23211774 0.23211774 ... 0.23860803 0.23860803 0.23860803]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.23596433 0.23596433 0.23596433 ... 0.26915005 0.26915005 0.26915005]\n",
      " [0.23651654 0.23651654 0.23651654 ... 0.23894407 0.23894407 0.23894407]\n",
      " [0.25193548 0.25193548 0.25193548 ... 0.24169211 0.24169211 0.24169211]\n",
      " ...\n",
      " [0.23852658 0.23852658 0.23852658 ... 0.2467216  0.2467216  0.2467216 ]\n",
      " [0.23997566 0.23997566 0.23997566 ... 0.23939347 0.23939347 0.23939347]\n",
      " [0.24100909 0.24100909 0.24100909 ... 0.24826053 0.24826053 0.24826053]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.2492095  0.2492095  0.2492095  ... 0.2430726  0.2430726  0.2430726 ]\n",
      " [0.24780393 0.24780393 0.24780393 ... 0.25046003 0.25046003 0.25046003]\n",
      " [0.23925593 0.23925593 0.23925593 ... 0.24441552 0.24441552 0.24441552]\n",
      " ...\n",
      " [0.24437857 0.24437857 0.24437857 ... 0.22597489 0.22597489 0.22597489]\n",
      " [0.24774449 0.24774449 0.24774449 ... 0.22904724 0.22904724 0.22904724]\n",
      " [0.2470679  0.2470679  0.2470679  ... 0.23571664 0.23571664 0.23571664]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.24168205 0.24168205 0.24168205 ... 0.2449967  0.2449967  0.2449967 ]\n",
      " [0.24563141 0.24563141 0.24563141 ... 0.23900276 0.23900276 0.23900276]\n",
      " [0.24054465 0.24054465 0.24054465 ... 0.23492447 0.23492447 0.23492447]\n",
      " ...\n",
      " [0.24272196 0.24272196 0.24272196 ... 0.23233636 0.23233636 0.23233636]\n",
      " [0.23745611 0.23745611 0.23745611 ... 0.22639707 0.22639707 0.22639707]\n",
      " [0.23723419 0.23723419 0.23723419 ... 0.23809229 0.23809229 0.23809229]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.24359247 0.24359247 0.24359247 ... 0.2547513  0.2547513  0.2547513 ]\n",
      " [0.23819195 0.23819195 0.23819195 ... 0.24763405 0.24763405 0.24763405]\n",
      " [0.2404322  0.2404322  0.2404322  ... 0.23285365 0.23285365 0.23285365]\n",
      " ...\n",
      " [0.24348113 0.24348113 0.24348113 ... 0.24994089 0.24994089 0.24994089]\n",
      " [0.24663773 0.24663773 0.24663773 ... 0.23768836 0.23768836 0.23768836]\n",
      " [0.24651194 0.24651194 0.24651194 ... 0.24051309 0.24051309 0.24051309]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.24335761 0.24335761 0.24335761 ... 0.24317142 0.24317142 0.24317142]\n",
      " [0.23848212 0.23848212 0.23848212 ... 0.24084581 0.24084581 0.24084581]\n",
      " [0.23708813 0.23708813 0.23708813 ... 0.23540488 0.23540488 0.23540488]\n",
      " ...\n",
      " [0.24609327 0.24609327 0.24609327 ... 0.23882979 0.23882979 0.23882979]\n",
      " [0.24759832 0.24759832 0.24759832 ... 0.2404404  0.2404404  0.2404404 ]\n",
      " [0.23936948 0.23936948 0.23936948 ... 0.23807251 0.23807251 0.23807251]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.24063134 0.24063134 0.24063134 ... 0.25067866 0.25067866 0.25067866]\n",
      " [0.2342376  0.2342376  0.2342376  ... 0.23996583 0.23996583 0.23996583]\n",
      " [0.24377349 0.24377349 0.24377349 ... 0.24956354 0.24956354 0.24956354]\n",
      " ...\n",
      " [0.23434904 0.23434904 0.23434904 ... 0.24960469 0.24960469 0.24960469]\n",
      " [0.23540778 0.23540778 0.23540778 ... 0.2592787  0.2592787  0.2592787 ]\n",
      " [0.24863966 0.24863966 0.24863966 ... 0.23813975 0.23813975 0.23813975]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.2450856  0.2450856  0.2450856  ... 0.23936898 0.23936898 0.23936898]\n",
      " [0.24244337 0.24244337 0.24244337 ... 0.23749949 0.23749949 0.23749949]\n",
      " [0.23950419 0.23950419 0.23950419 ... 0.24051969 0.24051969 0.24051969]\n",
      " ...\n",
      " [0.24793908 0.24793908 0.24793908 ... 0.23112749 0.23112749 0.23112749]\n",
      " [0.24537706 0.24537706 0.24537706 ... 0.2471132  0.2471132  0.2471132 ]\n",
      " [0.24537641 0.24537641 0.24537641 ... 0.23916289 0.23916289 0.23916289]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  9 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.16157222 0.16157222 0.16157222 ... 0.16512923 0.16512923 0.16512923]\n",
      " [0.17077649 0.17077649 0.17077649 ... 0.16518483 0.16518483 0.16518483]\n",
      " [0.16769904 0.16769904 0.16769904 ... 0.1654256  0.1654256  0.1654256 ]\n",
      " ...\n",
      " [0.16069056 0.16069056 0.16069056 ... 0.16277106 0.16277106 0.16277106]\n",
      " [0.16398567 0.16398567 0.16398567 ... 0.17398588 0.17398588 0.17398588]\n",
      " [0.15237144 0.15237144 0.15237144 ... 0.16489398 0.16489398 0.16489398]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.18537274 0.18537274 0.18537274 ... 0.17068884 0.17068884 0.17068884]\n",
      " [0.16184442 0.16184442 0.16184442 ... 0.16080418 0.16080418 0.16080418]\n",
      " [0.16586392 0.16586392 0.16586392 ... 0.1633567  0.1633567  0.1633567 ]\n",
      " ...\n",
      " [0.14759749 0.14759749 0.14759749 ... 0.1542436  0.1542436  0.1542436 ]\n",
      " [0.16890895 0.16890895 0.16890895 ... 0.15605208 0.15605208 0.15605208]\n",
      " [0.15591422 0.15591422 0.15591422 ... 0.16854218 0.16854218 0.16854218]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.16847657 0.16847657 0.16847657 ... 0.1634539  0.1634539  0.1634539 ]\n",
      " [0.15841576 0.15841576 0.15841576 ... 0.16098613 0.16098613 0.16098613]\n",
      " [0.17080072 0.17080072 0.17080072 ... 0.15592238 0.15592238 0.15592238]\n",
      " ...\n",
      " [0.15962985 0.15962985 0.15962985 ... 0.17550497 0.17550497 0.17550497]\n",
      " [0.15315077 0.15315077 0.15315077 ... 0.15666765 0.15666765 0.15666765]\n",
      " [0.15418318 0.15418318 0.15418318 ... 0.16544467 0.16544467 0.16544467]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.16310929 0.16310929 0.16310929 ... 0.1544624  0.1544624  0.1544624 ]\n",
      " [0.15214697 0.15214697 0.15214697 ... 0.16229007 0.16229007 0.16229007]\n",
      " [0.15214434 0.15214434 0.15214434 ... 0.1644713  0.1644713  0.1644713 ]\n",
      " ...\n",
      " [0.16631207 0.16631207 0.16631207 ... 0.16234872 0.16234872 0.16234872]\n",
      " [0.16226645 0.16226645 0.16226645 ... 0.16238758 0.16238758 0.16238758]\n",
      " [0.1599231  0.1599231  0.1599231  ... 0.15507168 0.15507168 0.15507168]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.14957887 0.14957887 0.14957887 ... 0.16962932 0.16962932 0.16962932]\n",
      " [0.17070949 0.17070949 0.17070949 ... 0.16172887 0.16172887 0.16172887]\n",
      " [0.15268046 0.15268046 0.15268046 ... 0.16460656 0.16460656 0.16460656]\n",
      " ...\n",
      " [0.14828192 0.14828192 0.14828192 ... 0.15969048 0.15969048 0.15969048]\n",
      " [0.16644925 0.16644925 0.16644925 ... 0.1633873  0.1633873  0.1633873 ]\n",
      " [0.15032679 0.15032679 0.15032679 ... 0.16038494 0.16038494 0.16038494]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.15231377 0.15231377 0.15231377 ... 0.16065633 0.16065633 0.16065633]\n",
      " [0.16685888 0.16685888 0.16685888 ... 0.15033077 0.15033077 0.15033077]\n",
      " [0.16064364 0.16064364 0.16064364 ... 0.16841339 0.16841339 0.16841339]\n",
      " ...\n",
      " [0.15547743 0.15547743 0.15547743 ... 0.16413298 0.16413298 0.16413298]\n",
      " [0.15736765 0.15736765 0.15736765 ... 0.17009449 0.17009449 0.17009449]\n",
      " [0.16228251 0.16228251 0.16228251 ... 0.15430254 0.15430254 0.15430254]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.15485911 0.15485911 0.15485911 ... 0.16386947 0.16386947 0.16386947]\n",
      " [0.16718489 0.16718489 0.16718489 ... 0.17222929 0.17222929 0.17222929]\n",
      " [0.1682381  0.1682381  0.1682381  ... 0.16738719 0.16738719 0.16738719]\n",
      " ...\n",
      " [0.13899252 0.13899252 0.13899252 ... 0.15748742 0.15748742 0.15748742]\n",
      " [0.15865462 0.15865462 0.15865462 ... 0.15726456 0.15726456 0.15726456]\n",
      " [0.16089061 0.16089061 0.16089061 ... 0.15897769 0.15897769 0.15897769]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.16008411 0.16008411 0.16008411 ... 0.16544764 0.16544764 0.16544764]\n",
      " [0.19030514 0.19030514 0.19030514 ... 0.1651603  0.1651603  0.1651603 ]\n",
      " [0.16454315 0.16454315 0.16454315 ... 0.1616449  0.1616449  0.1616449 ]\n",
      " ...\n",
      " [0.16749687 0.16749687 0.16749687 ... 0.15404643 0.15404643 0.15404643]\n",
      " [0.16488484 0.16488484 0.16488484 ... 0.15123181 0.15123181 0.15123181]\n",
      " [0.16255051 0.16255051 0.16255051 ... 0.14695789 0.14695789 0.14695789]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  10 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.10003712 0.10003712 0.10003712 ... 0.08497815 0.08497815 0.08497815]\n",
      " [0.09688903 0.09688903 0.09688903 ... 0.09604992 0.09604992 0.09604992]\n",
      " [0.12319525 0.12319525 0.12319525 ... 0.09762315 0.09762315 0.09762315]\n",
      " ...\n",
      " [0.08745632 0.08745632 0.08745632 ... 0.09535133 0.09535133 0.09535133]\n",
      " [0.07679282 0.07679282 0.07679282 ... 0.09122678 0.09122678 0.09122678]\n",
      " [0.10294555 0.10294555 0.10294555 ... 0.08727874 0.08727874 0.08727874]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.07678989 0.07678989 0.07678989 ... 0.09133281 0.09133281 0.09133281]\n",
      " [0.08378556 0.08378556 0.08378556 ... 0.08894291 0.08894291 0.08894291]\n",
      " [0.09577456 0.09577456 0.09577456 ... 0.0704622  0.0704622  0.0704622 ]\n",
      " ...\n",
      " [0.08958237 0.08958237 0.08958237 ... 0.08320993 0.08320993 0.08320993]\n",
      " [0.10054823 0.10054823 0.10054823 ... 0.10390186 0.10390186 0.10390186]\n",
      " [0.08451898 0.08451898 0.08451898 ... 0.08899355 0.08899355 0.08899355]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.1009915  0.1009915  0.1009915  ... 0.09913027 0.09913027 0.09913027]\n",
      " [0.0873951  0.0873951  0.0873951  ... 0.0866773  0.0866773  0.0866773 ]\n",
      " [0.08602159 0.08602159 0.08602159 ... 0.10274892 0.10274892 0.10274892]\n",
      " ...\n",
      " [0.09805566 0.09805566 0.09805566 ... 0.12085095 0.12085095 0.12085095]\n",
      " [0.08132933 0.08132933 0.08132933 ... 0.1042155  0.1042155  0.1042155 ]\n",
      " [0.09820689 0.09820689 0.09820689 ... 0.09371818 0.09371818 0.09371818]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.08470812 0.08470812 0.08470812 ... 0.0943158  0.0943158  0.0943158 ]\n",
      " [0.1045471  0.1045471  0.1045471  ... 0.08179934 0.08179934 0.08179934]\n",
      " [0.0846822  0.0846822  0.0846822  ... 0.09712088 0.09712088 0.09712088]\n",
      " ...\n",
      " [0.08590912 0.08590912 0.08590912 ... 0.10528032 0.10528032 0.10528032]\n",
      " [0.07960083 0.07960083 0.07960083 ... 0.07341883 0.07341883 0.07341883]\n",
      " [0.08401197 0.08401197 0.08401197 ... 0.09133281 0.09133281 0.09133281]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.08729307 0.08729307 0.08729307 ... 0.09458585 0.09458585 0.09458585]\n",
      " [0.10275536 0.10275536 0.10275536 ... 0.07493723 0.07493723 0.07493723]\n",
      " [0.08019235 0.08019235 0.08019235 ... 0.07518224 0.07518224 0.07518224]\n",
      " ...\n",
      " [0.07272655 0.07272655 0.07272655 ... 0.08189154 0.08189154 0.08189154]\n",
      " [0.10092846 0.10092846 0.10092846 ... 0.10226737 0.10226737 0.10226737]\n",
      " [0.08839656 0.08839656 0.08839656 ... 0.0857601  0.0857601  0.0857601 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.0856868  0.0856868  0.0856868  ... 0.08525398 0.08525398 0.08525398]\n",
      " [0.08684047 0.08684047 0.08684047 ... 0.08442767 0.08442767 0.08442767]\n",
      " [0.09864002 0.09864002 0.09864002 ... 0.0891373  0.0891373  0.0891373 ]\n",
      " ...\n",
      " [0.0917903  0.0917903  0.0917903  ... 0.10720593 0.10720593 0.10720593]\n",
      " [0.08041435 0.08041435 0.08041435 ... 0.11564782 0.11564782 0.11564782]\n",
      " [0.0889111  0.0889111  0.0889111  ... 0.10748225 0.10748225 0.10748225]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[0.07051595 0.07051595 0.07051595 ... 0.09863832 0.09863832 0.09863832]\n",
      " [0.10151549 0.10151549 0.10151549 ... 0.08745562 0.08745562 0.08745562]\n",
      " [0.08284773 0.08284773 0.08284773 ... 0.0894253  0.0894253  0.0894253 ]\n",
      " ...\n",
      " [0.09273824 0.09273824 0.09273824 ... 0.10362407 0.10362407 0.10362407]\n",
      " [0.07717305 0.07717305 0.07717305 ... 0.07185896 0.07185896 0.07185896]\n",
      " [0.08902808 0.08902808 0.08902808 ... 0.08645738 0.08645738 0.08645738]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.09181754 0.09181754 0.09181754 ... 0.09161401 0.09161401 0.09161401]\n",
      " [0.08743183 0.08743183 0.08743183 ... 0.09396504 0.09396504 0.09396504]\n",
      " [0.09034974 0.09034974 0.09034974 ... 0.08850408 0.08850408 0.08850408]\n",
      " ...\n",
      " [0.09409267 0.09409267 0.09409267 ... 0.09516071 0.09516071 0.09516071]\n",
      " [0.08361455 0.08361455 0.08361455 ... 0.09440892 0.09440892 0.09440892]\n",
      " [0.08900949 0.08900949 0.08900949 ... 0.09012158 0.09012158 0.09012158]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  11 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.02893063 0.02893063 0.02893063 ... 0.03236566 0.03236566 0.03236566]\n",
      " [0.02894585 0.02894585 0.02894585 ... 0.02948609 0.02948609 0.02948609]\n",
      " [0.03007779 0.03007779 0.03007779 ... 0.03538378 0.03538378 0.03538378]\n",
      " ...\n",
      " [0.00219986 0.00219986 0.00219986 ... 0.02947497 0.02947497 0.02947497]\n",
      " [0.02147031 0.02147031 0.02147031 ... 0.03643508 0.03643508 0.03643508]\n",
      " [0.02589352 0.02589352 0.02589352 ... 0.04096774 0.04096774 0.04096774]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.01766787 0.01766787 0.01766787 ... 0.01927358 0.01927358 0.01927358]\n",
      " [0.03306589 0.03306589 0.03306589 ... 0.02296352 0.02296352 0.02296352]\n",
      " [0.02531921 0.02531921 0.02531921 ... 0.01665077 0.01665077 0.01665077]\n",
      " ...\n",
      " [0.04502959 0.04502959 0.04502959 ... 0.01226062 0.01226062 0.01226062]\n",
      " [0.03111013 0.03111013 0.03111013 ... 0.02834061 0.02834061 0.02834061]\n",
      " [0.0145384  0.0145384  0.0145384  ... 0.03410066 0.03410066 0.03410066]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.00329275 0.00329275 0.00329275 ... 0.03590655 0.03590655 0.03590655]\n",
      " [0.02090476 0.02090476 0.02090476 ... 0.02845512 0.02845512 0.02845512]\n",
      " [0.02724263 0.02724263 0.02724263 ... 0.0185226  0.0185226  0.0185226 ]\n",
      " ...\n",
      " [0.02595758 0.02595758 0.02595758 ... 0.01140036 0.01140036 0.01140036]\n",
      " [0.02402123 0.02402123 0.02402123 ... 0.03599446 0.03599446 0.03599446]\n",
      " [0.02116521 0.02116521 0.02116521 ... 0.01938434 0.01938434 0.01938434]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[ 0.0071637   0.0071637   0.0071637  ...  0.02560886  0.02560886\n",
      "   0.02560886]\n",
      " [ 0.04184821  0.04184821  0.04184821 ...  0.02091652  0.02091652\n",
      "   0.02091652]\n",
      " [ 0.02917544  0.02917544  0.02917544 ...  0.03314517  0.03314517\n",
      "   0.03314517]\n",
      " ...\n",
      " [ 0.02236191  0.02236191  0.02236191 ... -0.00237408 -0.00237408\n",
      "  -0.00237408]\n",
      " [ 0.03187193  0.03187193  0.03187193 ...  0.03512165  0.03512165\n",
      "   0.03512165]\n",
      " [ 0.02109879  0.02109879  0.02109879 ...  0.03209988  0.03209988\n",
      "   0.03209988]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[0.02160804 0.02160804 0.02160804 ... 0.04097208 0.04097208 0.04097208]\n",
      " [0.03441573 0.03441573 0.03441573 ... 0.02942595 0.02942595 0.02942595]\n",
      " [0.02509802 0.02509802 0.02509802 ... 0.02528129 0.02528129 0.02528129]\n",
      " ...\n",
      " [0.00901338 0.00901338 0.00901338 ... 0.01968316 0.01968316 0.01968316]\n",
      " [0.01973188 0.01973188 0.01973188 ... 0.02388313 0.02388313 0.02388313]\n",
      " [0.00474547 0.00474547 0.00474547 ... 0.01859307 0.01859307 0.01859307]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[0.03566417 0.03566417 0.03566417 ... 0.02753892 0.02753892 0.02753892]\n",
      " [0.04943541 0.04943541 0.04943541 ... 0.02519118 0.02519118 0.02519118]\n",
      " [0.04118384 0.04118384 0.04118384 ... 0.02523399 0.02523399 0.02523399]\n",
      " ...\n",
      " [0.01922578 0.01922578 0.01922578 ... 0.03575213 0.03575213 0.03575213]\n",
      " [0.03280402 0.03280402 0.03280402 ... 0.03679773 0.03679773 0.03679773]\n",
      " [0.02703141 0.02703141 0.02703141 ... 0.00917346 0.00917346 0.00917346]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[ 0.0235071   0.0235071   0.0235071  ...  0.03314642  0.03314642\n",
      "   0.03314642]\n",
      " [-0.00083015 -0.00083015 -0.00083015 ...  0.02196993  0.02196993\n",
      "   0.02196993]\n",
      " [ 0.04389133  0.04389133  0.04389133 ...  0.0229232   0.0229232\n",
      "   0.0229232 ]\n",
      " ...\n",
      " [ 0.01123112  0.01123112  0.01123112 ...  0.01692081  0.01692081\n",
      "   0.01692081]\n",
      " [ 0.01698994  0.01698994  0.01698994 ...  0.03008593  0.03008593\n",
      "   0.03008593]\n",
      " [ 0.02875954  0.02875954  0.02875954 ...  0.04631732  0.04631732\n",
      "   0.04631732]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[0.04388897 0.04388897 0.04388897 ... 0.03464395 0.03464395 0.03464395]\n",
      " [0.03777764 0.03777764 0.03777764 ... 0.04365015 0.04365015 0.04365015]\n",
      " [0.01443098 0.01443098 0.01443098 ... 0.02218361 0.02218361 0.02218361]\n",
      " ...\n",
      " [0.02921516 0.02921516 0.02921516 ... 0.03624842 0.03624842 0.03624842]\n",
      " [0.02695499 0.02695499 0.02695499 ... 0.03741177 0.03741177 0.03741177]\n",
      " [0.03670634 0.03670634 0.03670634 ... 0.03013951 0.03013951 0.03013951]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  12 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.02505599 -0.02505599 -0.02505599 ... -0.02554284 -0.02554284\n",
      "  -0.02554284]\n",
      " [-0.02333529 -0.02333529 -0.02333529 ... -0.03851294 -0.03851294\n",
      "  -0.03851294]\n",
      " [-0.03780506 -0.03780506 -0.03780506 ... -0.03587885 -0.03587885\n",
      "  -0.03587885]\n",
      " ...\n",
      " [-0.02959952 -0.02959952 -0.02959952 ... -0.02874575 -0.02874575\n",
      "  -0.02874575]\n",
      " [-0.03088887 -0.03088887 -0.03088887 ... -0.02098713 -0.02098713\n",
      "  -0.02098713]\n",
      " [-0.03072194 -0.03072194 -0.03072194 ... -0.02162267 -0.02162267\n",
      "  -0.02162267]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.03369994 -0.03369994 -0.03369994 ... -0.02528584 -0.02528584\n",
      "  -0.02528584]\n",
      " [-0.03591504 -0.03591504 -0.03591504 ... -0.00925754 -0.00925754\n",
      "  -0.00925754]\n",
      " [-0.02865777 -0.02865777 -0.02865777 ... -0.03381521 -0.03381521\n",
      "  -0.03381521]\n",
      " ...\n",
      " [-0.03260084 -0.03260084 -0.03260084 ... -0.0396219  -0.0396219\n",
      "  -0.0396219 ]\n",
      " [-0.01958488 -0.01958488 -0.01958488 ... -0.05303847 -0.05303847\n",
      "  -0.05303847]\n",
      " [-0.03719386 -0.03719386 -0.03719386 ... -0.03719319 -0.03719319\n",
      "  -0.03719319]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.05073925 -0.05073925 -0.05073925 ... -0.02731051 -0.02731051\n",
      "  -0.02731051]\n",
      " [-0.02546409 -0.02546409 -0.02546409 ... -0.05753501 -0.05753501\n",
      "  -0.05753501]\n",
      " [-0.01806971 -0.01806971 -0.01806971 ... -0.03385238 -0.03385238\n",
      "  -0.03385238]\n",
      " ...\n",
      " [-0.03415452 -0.03415452 -0.03415452 ... -0.03517817 -0.03517817\n",
      "  -0.03517817]\n",
      " [-0.02754249 -0.02754249 -0.02754249 ... -0.04220095 -0.04220095\n",
      "  -0.04220095]\n",
      " [-0.03872375 -0.03872375 -0.03872375 ... -0.033874   -0.033874\n",
      "  -0.033874  ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.03076304 -0.03076304 -0.03076304 ... -0.02377994 -0.02377994\n",
      "  -0.02377994]\n",
      " [-0.04562377 -0.04562377 -0.04562377 ... -0.03612421 -0.03612421\n",
      "  -0.03612421]\n",
      " [-0.04043124 -0.04043124 -0.04043124 ... -0.01731583 -0.01731583\n",
      "  -0.01731583]\n",
      " ...\n",
      " [-0.05065624 -0.05065624 -0.05065624 ... -0.0267981  -0.0267981\n",
      "  -0.0267981 ]\n",
      " [-0.03196561 -0.03196561 -0.03196561 ... -0.02401988 -0.02401988\n",
      "  -0.02401988]\n",
      " [-0.03553066 -0.03553066 -0.03553066 ... -0.03052551 -0.03052551\n",
      "  -0.03052551]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.01795654 -0.01795654 -0.01795654 ... -0.02245413 -0.02245413\n",
      "  -0.02245413]\n",
      " [-0.04761869 -0.04761869 -0.04761869 ... -0.02521348 -0.02521348\n",
      "  -0.02521348]\n",
      " [-0.02598454 -0.02598454 -0.02598454 ... -0.03706016 -0.03706016\n",
      "  -0.03706016]\n",
      " ...\n",
      " [-0.02397939 -0.02397939 -0.02397939 ... -0.02590064 -0.02590064\n",
      "  -0.02590064]\n",
      " [-0.03823148 -0.03823148 -0.03823148 ... -0.03758223 -0.03758223\n",
      "  -0.03758223]\n",
      " [-0.03708002 -0.03708002 -0.03708002 ... -0.02960684 -0.02960684\n",
      "  -0.02960684]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.04002214 -0.04002214 -0.04002214 ... -0.03563115 -0.03563115\n",
      "  -0.03563115]\n",
      " [-0.03090621 -0.03090621 -0.03090621 ... -0.02554284 -0.02554284\n",
      "  -0.02554284]\n",
      " [-0.01902784 -0.01902784 -0.01902784 ... -0.02036298 -0.02036298\n",
      "  -0.02036298]\n",
      " ...\n",
      " [-0.02133322 -0.02133322 -0.02133322 ... -0.020066   -0.020066\n",
      "  -0.020066  ]\n",
      " [-0.03346303 -0.03346303 -0.03346303 ... -0.05530505 -0.05530505\n",
      "  -0.05530505]\n",
      " [-0.01539387 -0.01539387 -0.01539387 ... -0.02332284 -0.02332284\n",
      "  -0.02332284]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.05761691 -0.05761691 -0.05761691 ... -0.02283629 -0.02283629\n",
      "  -0.02283629]\n",
      " [ 0.00010751  0.00010751  0.00010751 ... -0.02102854 -0.02102854\n",
      "  -0.02102854]\n",
      " [-0.04378376 -0.04378376 -0.04378376 ... -0.04107352 -0.04107352\n",
      "  -0.04107352]\n",
      " ...\n",
      " [-0.04339428 -0.04339428 -0.04339428 ... -0.02312767 -0.02312767\n",
      "  -0.02312767]\n",
      " [-0.04314069 -0.04314069 -0.04314069 ... -0.01774563 -0.01774563\n",
      "  -0.01774563]\n",
      " [-0.0241443  -0.0241443  -0.0241443  ... -0.0352963  -0.0352963\n",
      "  -0.0352963 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.04098557 -0.04098557 -0.04098557 ... -0.02789271 -0.02789271\n",
      "  -0.02789271]\n",
      " [-0.02983604 -0.02983604 -0.02983604 ... -0.02811075 -0.02811075\n",
      "  -0.02811075]\n",
      " [-0.02167343 -0.02167343 -0.02167343 ... -0.01649129 -0.01649129\n",
      "  -0.01649129]\n",
      " ...\n",
      " [-0.03608435 -0.03608435 -0.03608435 ... -0.02980741 -0.02980741\n",
      "  -0.02980741]\n",
      " [-0.02650069 -0.02650069 -0.02650069 ... -0.02782847 -0.02782847\n",
      "  -0.02782847]\n",
      " [-0.03980462 -0.03980462 -0.03980462 ... -0.01936164 -0.01936164\n",
      "  -0.01936164]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  13 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.08359683 -0.08359683 -0.08359683 ... -0.08795975 -0.08795975\n",
      "  -0.08795975]\n",
      " [-0.09024791 -0.09024791 -0.09024791 ... -0.05053963 -0.05053963\n",
      "  -0.05053963]\n",
      " [-0.07120103 -0.07120103 -0.07120103 ... -0.0837634  -0.0837634\n",
      "  -0.0837634 ]\n",
      " ...\n",
      " [-0.06049202 -0.06049202 -0.06049202 ... -0.06567349 -0.06567349\n",
      "  -0.06567349]\n",
      " [-0.06025257 -0.06025257 -0.06025257 ... -0.10210505 -0.10210505\n",
      "  -0.10210505]\n",
      " [-0.09353594 -0.09353594 -0.09353594 ... -0.09660237 -0.09660237\n",
      "  -0.09660237]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.08240079 -0.08240079 -0.08240079 ... -0.05399724 -0.05399724\n",
      "  -0.05399724]\n",
      " [-0.08359031 -0.08359031 -0.08359031 ... -0.09802179 -0.09802179\n",
      "  -0.09802179]\n",
      " [-0.08766444 -0.08766444 -0.08766444 ... -0.09026612 -0.09026612\n",
      "  -0.09026612]\n",
      " ...\n",
      " [-0.09843458 -0.09843458 -0.09843458 ... -0.11011045 -0.11011045\n",
      "  -0.11011045]\n",
      " [-0.07641515 -0.07641515 -0.07641515 ... -0.06824362 -0.06824362\n",
      "  -0.06824362]\n",
      " [-0.070493   -0.070493   -0.070493   ... -0.07669139 -0.07669139\n",
      "  -0.07669139]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.09147602 -0.09147602 -0.09147602 ... -0.07438971 -0.07438971\n",
      "  -0.07438971]\n",
      " [-0.07696377 -0.07696377 -0.07696377 ... -0.073581   -0.073581\n",
      "  -0.073581  ]\n",
      " [-0.0782864  -0.0782864  -0.0782864  ... -0.07438721 -0.07438721\n",
      "  -0.07438721]\n",
      " ...\n",
      " [-0.08699076 -0.08699076 -0.08699076 ... -0.07934695 -0.07934695\n",
      "  -0.07934695]\n",
      " [-0.0893309  -0.0893309  -0.0893309  ... -0.07238627 -0.07238627\n",
      "  -0.07238627]\n",
      " [-0.08404531 -0.08404531 -0.08404531 ... -0.09115787 -0.09115787\n",
      "  -0.09115787]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.09359051 -0.09359051 -0.09359051 ... -0.07992933 -0.07992933\n",
      "  -0.07992933]\n",
      " [-0.08776441 -0.08776441 -0.08776441 ... -0.08589374 -0.08589374\n",
      "  -0.08589374]\n",
      " [-0.0751693  -0.0751693  -0.0751693  ... -0.07459015 -0.07459015\n",
      "  -0.07459015]\n",
      " ...\n",
      " [-0.07004195 -0.07004195 -0.07004195 ... -0.06111808 -0.06111808\n",
      "  -0.06111808]\n",
      " [-0.07545707 -0.07545707 -0.07545707 ... -0.07968441 -0.07968441\n",
      "  -0.07968441]\n",
      " [-0.05995555 -0.05995555 -0.05995555 ... -0.06889305 -0.06889305\n",
      "  -0.06889305]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.08655699 -0.08655699 -0.08655699 ... -0.07855822 -0.07855822\n",
      "  -0.07855822]\n",
      " [-0.08882418 -0.08882418 -0.08882418 ... -0.07017896 -0.07017896\n",
      "  -0.07017896]\n",
      " [-0.07729378 -0.07729378 -0.07729378 ... -0.11031011 -0.11031011\n",
      "  -0.11031011]\n",
      " ...\n",
      " [-0.08701098 -0.08701098 -0.08701098 ... -0.0817959  -0.0817959\n",
      "  -0.0817959 ]\n",
      " [-0.07063848 -0.07063848 -0.07063848 ... -0.06965432 -0.06965432\n",
      "  -0.06965432]\n",
      " [-0.07676083 -0.07676083 -0.07676083 ... -0.08528998 -0.08528998\n",
      "  -0.08528998]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.0804951  -0.0804951  -0.0804951  ... -0.0840365  -0.0840365\n",
      "  -0.0840365 ]\n",
      " [-0.07702969 -0.07702969 -0.07702969 ... -0.07325967 -0.07325967\n",
      "  -0.07325967]\n",
      " [-0.07688551 -0.07688551 -0.07688551 ... -0.05552104 -0.05552104\n",
      "  -0.05552104]\n",
      " ...\n",
      " [-0.06456528 -0.06456528 -0.06456528 ... -0.05036871 -0.05036871\n",
      "  -0.05036871]\n",
      " [-0.080868   -0.080868   -0.080868   ... -0.09784247 -0.09784247\n",
      "  -0.09784247]\n",
      " [-0.0797004  -0.0797004  -0.0797004  ... -0.05132641 -0.05132641\n",
      "  -0.05132641]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.08475966 -0.08475966 -0.08475966 ... -0.07139061 -0.07139061\n",
      "  -0.07139061]\n",
      " [-0.07996359 -0.07996359 -0.07996359 ... -0.0731626  -0.0731626\n",
      "  -0.0731626 ]\n",
      " [-0.08431602 -0.08431602 -0.08431602 ... -0.07770711 -0.07770711\n",
      "  -0.07770711]\n",
      " ...\n",
      " [-0.07300666 -0.07300666 -0.07300666 ... -0.06386033 -0.06386033\n",
      "  -0.06386033]\n",
      " [-0.08452459 -0.08452459 -0.08452459 ... -0.07496235 -0.07496235\n",
      "  -0.07496235]\n",
      " [-0.1058958  -0.1058958  -0.1058958  ... -0.07914495 -0.07914495\n",
      "  -0.07914495]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.07937467 -0.07937467 -0.07937467 ... -0.10208274 -0.10208274\n",
      "  -0.10208274]\n",
      " [-0.09160718 -0.09160718 -0.09160718 ... -0.08510345 -0.08510345\n",
      "  -0.08510345]\n",
      " [-0.07330316 -0.07330316 -0.07330316 ... -0.06143421 -0.06143421\n",
      "  -0.06143421]\n",
      " ...\n",
      " [-0.10011184 -0.10011184 -0.10011184 ... -0.0938431  -0.0938431\n",
      "  -0.0938431 ]\n",
      " [-0.09644696 -0.09644696 -0.09644696 ... -0.08332084 -0.08332084\n",
      "  -0.08332084]\n",
      " [-0.08130075 -0.08130075 -0.08130075 ... -0.08048278 -0.08048278\n",
      "  -0.08048278]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  14 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13518184 -0.13518184 -0.13518184 ... -0.12721187 -0.12721187\n",
      "  -0.12721187]\n",
      " [-0.11077808 -0.11077808 -0.11077808 ... -0.12843534 -0.12843534\n",
      "  -0.12843534]\n",
      " [-0.10311799 -0.10311799 -0.10311799 ... -0.13690932 -0.13690932\n",
      "  -0.13690932]\n",
      " ...\n",
      " [-0.13113362 -0.13113362 -0.13113362 ... -0.13058433 -0.13058433\n",
      "  -0.13058433]\n",
      " [-0.13294964 -0.13294964 -0.13294964 ... -0.14016397 -0.14016397\n",
      "  -0.14016397]\n",
      " [-0.12572125 -0.12572125 -0.12572125 ... -0.13133661 -0.13133661\n",
      "  -0.13133661]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1087272  -0.1087272  -0.1087272  ... -0.14244205 -0.14244205\n",
      "  -0.14244205]\n",
      " [-0.13119288 -0.13119288 -0.13119288 ... -0.13623011 -0.13623011\n",
      "  -0.13623011]\n",
      " [-0.1257813  -0.1257813  -0.1257813  ... -0.1191218  -0.1191218\n",
      "  -0.1191218 ]\n",
      " ...\n",
      " [-0.1338479  -0.1338479  -0.1338479  ... -0.15440923 -0.15440923\n",
      "  -0.15440923]\n",
      " [-0.12871653 -0.12871653 -0.12871653 ... -0.12879038 -0.12879038\n",
      "  -0.12879038]\n",
      " [-0.12648495 -0.12648495 -0.12648495 ... -0.11879852 -0.11879852\n",
      "  -0.11879852]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14462689 -0.14462689 -0.14462689 ... -0.12132397 -0.12132397\n",
      "  -0.12132397]\n",
      " [-0.10794976 -0.10794976 -0.10794976 ... -0.11562745 -0.11562745\n",
      "  -0.11562745]\n",
      " [-0.12277856 -0.12277856 -0.12277856 ... -0.13558155 -0.13558155\n",
      "  -0.13558155]\n",
      " ...\n",
      " [-0.13525864 -0.13525864 -0.13525864 ... -0.14958382 -0.14958382\n",
      "  -0.14958382]\n",
      " [-0.12801409 -0.12801409 -0.12801409 ... -0.13574755 -0.13574755\n",
      "  -0.13574755]\n",
      " [-0.11680923 -0.11680923 -0.11680923 ... -0.11415781 -0.11415781\n",
      "  -0.11415781]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.13527994 -0.13527994 -0.13527994 ... -0.14361921 -0.14361921\n",
      "  -0.14361921]\n",
      " [-0.13244191 -0.13244191 -0.13244191 ... -0.13858463 -0.13858463\n",
      "  -0.13858463]\n",
      " [-0.12617591 -0.12617591 -0.12617591 ... -0.12715584 -0.12715584\n",
      "  -0.12715584]\n",
      " ...\n",
      " [-0.13019583 -0.13019583 -0.13019583 ... -0.12288261 -0.12288261\n",
      "  -0.12288261]\n",
      " [-0.12377787 -0.12377787 -0.12377787 ... -0.13208854 -0.13208854\n",
      "  -0.13208854]\n",
      " [-0.12286023 -0.12286023 -0.12286023 ... -0.13827734 -0.13827734\n",
      "  -0.13827734]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.13847612 -0.13847612 -0.13847612 ... -0.13840361 -0.13840361\n",
      "  -0.13840361]\n",
      " [-0.10033958 -0.10033958 -0.10033958 ... -0.13290228 -0.13290228\n",
      "  -0.13290228]\n",
      " [-0.12497595 -0.12497595 -0.12497595 ... -0.13946688 -0.13946688\n",
      "  -0.13946688]\n",
      " ...\n",
      " [-0.1379116  -0.1379116  -0.1379116  ... -0.12875578 -0.12875578\n",
      "  -0.12875578]\n",
      " [-0.1259566  -0.1259566  -0.1259566  ... -0.14191632 -0.14191632\n",
      "  -0.14191632]\n",
      " [-0.12631743 -0.12631743 -0.12631743 ... -0.11679738 -0.11679738\n",
      "  -0.11679738]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.14471094 -0.14471094 -0.14471094 ... -0.12637153 -0.12637153\n",
      "  -0.12637153]\n",
      " [-0.11228187 -0.11228187 -0.11228187 ... -0.13769113 -0.13769113\n",
      "  -0.13769113]\n",
      " [-0.1219594  -0.1219594  -0.1219594  ... -0.13774094 -0.13774094\n",
      "  -0.13774094]\n",
      " ...\n",
      " [-0.11156736 -0.11156736 -0.11156736 ... -0.14636087 -0.14636087\n",
      "  -0.14636087]\n",
      " [-0.10923035 -0.10923035 -0.10923035 ... -0.14770962 -0.14770962\n",
      "  -0.14770962]\n",
      " [-0.13432845 -0.13432845 -0.13432845 ... -0.13341948 -0.13341948\n",
      "  -0.13341948]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.12616777 -0.12616777 -0.12616777 ... -0.13521612 -0.13521612\n",
      "  -0.13521612]\n",
      " [-0.13357344 -0.13357344 -0.13357344 ... -0.11992443 -0.11992443\n",
      "  -0.11992443]\n",
      " [-0.11135595 -0.11135595 -0.11135595 ... -0.13597302 -0.13597302\n",
      "  -0.13597302]\n",
      " ...\n",
      " [-0.13160466 -0.13160466 -0.13160466 ... -0.13020729 -0.13020729\n",
      "  -0.13020729]\n",
      " [-0.11539046 -0.11539046 -0.11539046 ... -0.12409115 -0.12409115\n",
      "  -0.12409115]\n",
      " [-0.12360273 -0.12360273 -0.12360273 ... -0.14292052 -0.14292052\n",
      "  -0.14292052]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.12386709 -0.12386709 -0.12386709 ... -0.11824172 -0.11824172\n",
      "  -0.11824172]\n",
      " [-0.13777976 -0.13777976 -0.13777976 ... -0.13862632 -0.13862632\n",
      "  -0.13862632]\n",
      " [-0.11304717 -0.11304717 -0.11304717 ... -0.12511167 -0.12511167\n",
      "  -0.12511167]\n",
      " ...\n",
      " [-0.11957817 -0.11957817 -0.11957817 ... -0.10024902 -0.10024902\n",
      "  -0.10024902]\n",
      " [-0.10138877 -0.10138877 -0.10138877 ... -0.12496872 -0.12496872\n",
      "  -0.12496872]\n",
      " [-0.12761517 -0.12761517 -0.12761517 ... -0.1197378  -0.1197378\n",
      "  -0.1197378 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  15 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.15322235 -0.15322235 -0.15322235 ... -0.17870262 -0.17870262\n",
      "  -0.17870262]\n",
      " [-0.15843932 -0.15843932 -0.15843932 ... -0.17112374 -0.17112374\n",
      "  -0.17112374]\n",
      " [-0.14272389 -0.14272389 -0.14272389 ... -0.18848267 -0.18848267\n",
      "  -0.18848267]\n",
      " ...\n",
      " [-0.14168906 -0.14168906 -0.14168906 ... -0.17161945 -0.17161945\n",
      "  -0.17161945]\n",
      " [-0.16971883 -0.16971883 -0.16971883 ... -0.18747723 -0.18747723\n",
      "  -0.18747723]\n",
      " [-0.17587632 -0.17587632 -0.17587632 ... -0.14670241 -0.14670241\n",
      "  -0.14670241]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.17669049 -0.17669049 -0.17669049 ... -0.16980606 -0.16980606\n",
      "  -0.16980606]\n",
      " [-0.17796502 -0.17796502 -0.17796502 ... -0.17998312 -0.17998312\n",
      "  -0.17998312]\n",
      " [-0.18903658 -0.18903658 -0.18903658 ... -0.17862275 -0.17862275\n",
      "  -0.17862275]\n",
      " ...\n",
      " [-0.14882204 -0.14882204 -0.14882204 ... -0.17519255 -0.17519255\n",
      "  -0.17519255]\n",
      " [-0.17063296 -0.17063296 -0.17063296 ... -0.18968877 -0.18968877\n",
      "  -0.18968877]\n",
      " [-0.16892919 -0.16892919 -0.16892919 ... -0.20520917 -0.20520917\n",
      "  -0.20520917]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15722051 -0.15722051 -0.15722051 ... -0.14814909 -0.14814909\n",
      "  -0.14814909]\n",
      " [-0.1405144  -0.1405144  -0.1405144  ... -0.18730277 -0.18730277\n",
      "  -0.18730277]\n",
      " [-0.15212594 -0.15212594 -0.15212594 ... -0.16692024 -0.16692024\n",
      "  -0.16692024]\n",
      " ...\n",
      " [-0.1654624  -0.1654624  -0.1654624  ... -0.16364264 -0.16364264\n",
      "  -0.16364264]\n",
      " [-0.15102306 -0.15102306 -0.15102306 ... -0.15481718 -0.15481718\n",
      "  -0.15481718]\n",
      " [-0.16907454 -0.16907454 -0.16907454 ... -0.14893793 -0.14893793\n",
      "  -0.14893793]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18223503 -0.18223503 -0.18223503 ... -0.17907727 -0.17907727\n",
      "  -0.17907727]\n",
      " [-0.17862962 -0.17862962 -0.17862962 ... -0.1776942  -0.1776942\n",
      "  -0.1776942 ]\n",
      " [-0.15039985 -0.15039985 -0.15039985 ... -0.17032427 -0.17032427\n",
      "  -0.17032427]\n",
      " ...\n",
      " [-0.17873368 -0.17873368 -0.17873368 ... -0.18057495 -0.18057495\n",
      "  -0.18057495]\n",
      " [-0.17652112 -0.17652112 -0.17652112 ... -0.17861697 -0.17861697\n",
      "  -0.17861697]\n",
      " [-0.17571735 -0.17571735 -0.17571735 ... -0.16262111 -0.16262111\n",
      "  -0.16262111]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.16539043 -0.16539043 -0.16539043 ... -0.16610745 -0.16610745\n",
      "  -0.16610745]\n",
      " [-0.1808623  -0.1808623  -0.1808623  ... -0.18745434 -0.18745434\n",
      "  -0.18745434]\n",
      " [-0.16575058 -0.16575058 -0.16575058 ... -0.16297343 -0.16297343\n",
      "  -0.16297343]\n",
      " ...\n",
      " [-0.17534447 -0.17534447 -0.17534447 ... -0.18408328 -0.18408328\n",
      "  -0.18408328]\n",
      " [-0.1830801  -0.1830801  -0.1830801  ... -0.18424387 -0.18424387\n",
      "  -0.18424387]\n",
      " [-0.18605468 -0.18605468 -0.18605468 ... -0.14855367 -0.14855367\n",
      "  -0.14855367]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.16818431 -0.16818431 -0.16818431 ... -0.15261953 -0.15261953\n",
      "  -0.15261953]\n",
      " [-0.17357433 -0.17357433 -0.17357433 ... -0.17592761 -0.17592761\n",
      "  -0.17592761]\n",
      " [-0.14732794 -0.14732794 -0.14732794 ... -0.17775589 -0.17775589\n",
      "  -0.17775589]\n",
      " ...\n",
      " [-0.17481813 -0.17481813 -0.17481813 ... -0.17914939 -0.17914939\n",
      "  -0.17914939]\n",
      " [-0.18182197 -0.18182197 -0.18182197 ... -0.1786428  -0.1786428\n",
      "  -0.1786428 ]\n",
      " [-0.17214705 -0.17214705 -0.17214705 ... -0.19166464 -0.19166464\n",
      "  -0.19166464]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.19203764 -0.19203764 -0.19203764 ... -0.16889742 -0.16889742\n",
      "  -0.16889742]\n",
      " [-0.18539405 -0.18539405 -0.18539405 ... -0.16081278 -0.16081278\n",
      "  -0.16081278]\n",
      " [-0.15942746 -0.15942746 -0.15942746 ... -0.16775088 -0.16775088\n",
      "  -0.16775088]\n",
      " ...\n",
      " [-0.13168341 -0.13168341 -0.13168341 ... -0.16370103 -0.16370103\n",
      "  -0.16370103]\n",
      " [-0.1673247  -0.1673247  -0.1673247  ... -0.18096101 -0.18096101\n",
      "  -0.18096101]\n",
      " [-0.1520039  -0.1520039  -0.1520039  ... -0.20392978 -0.20392978\n",
      "  -0.20392978]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.16654153 -0.16654153 -0.16654153 ... -0.17599508 -0.17599508\n",
      "  -0.17599508]\n",
      " [-0.18551971 -0.18551971 -0.18551971 ... -0.16310258 -0.16310258\n",
      "  -0.16310258]\n",
      " [-0.1785169  -0.1785169  -0.1785169  ... -0.17386085 -0.17386085\n",
      "  -0.17386085]\n",
      " ...\n",
      " [-0.17902757 -0.17902757 -0.17902757 ... -0.2136758  -0.2136758\n",
      "  -0.2136758 ]\n",
      " [-0.18603005 -0.18603005 -0.18603005 ... -0.18101639 -0.18101639\n",
      "  -0.18101639]\n",
      " [-0.16305614 -0.16305614 -0.16305614 ... -0.18237886 -0.18237886\n",
      "  -0.18237886]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  16 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.22410838 -0.22410838 -0.22410838 ... -0.19928336 -0.19928336\n",
      "  -0.19928336]\n",
      " [-0.20469892 -0.20469892 -0.20469892 ... -0.19558935 -0.19558935\n",
      "  -0.19558935]\n",
      " [-0.19532439 -0.19532439 -0.19532439 ... -0.22929418 -0.22929418\n",
      "  -0.22929418]\n",
      " ...\n",
      " [-0.2157424  -0.2157424  -0.2157424  ... -0.19569314 -0.19569314\n",
      "  -0.19569314]\n",
      " [-0.21398807 -0.21398807 -0.21398807 ... -0.2135714  -0.2135714\n",
      "  -0.2135714 ]\n",
      " [-0.19648087 -0.19648087 -0.19648087 ... -0.23419078 -0.23419078\n",
      "  -0.23419078]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.20335314 -0.20335314 -0.20335314 ... -0.21006052 -0.21006052\n",
      "  -0.21006052]\n",
      " [-0.21110621 -0.21110621 -0.21110621 ... -0.24195579 -0.24195579\n",
      "  -0.24195579]\n",
      " [-0.21984792 -0.21984792 -0.21984792 ... -0.21060213 -0.21060213\n",
      "  -0.21060213]\n",
      " ...\n",
      " [-0.21619621 -0.21619621 -0.21619621 ... -0.21057378 -0.21057378\n",
      "  -0.21057378]\n",
      " [-0.19862103 -0.19862103 -0.19862103 ... -0.21056059 -0.21056059\n",
      "  -0.21056059]\n",
      " [-0.20201169 -0.20201169 -0.20201169 ... -0.20310447 -0.20310447\n",
      "  -0.20310447]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.22758219 -0.22758219 -0.22758219 ... -0.21262607 -0.21262607\n",
      "  -0.21262607]\n",
      " [-0.22089307 -0.22089307 -0.22089307 ... -0.213675   -0.213675\n",
      "  -0.213675  ]\n",
      " [-0.22713034 -0.22713034 -0.22713034 ... -0.18170337 -0.18170337\n",
      "  -0.18170337]\n",
      " ...\n",
      " [-0.19636613 -0.19636613 -0.19636613 ... -0.224623   -0.224623\n",
      "  -0.224623  ]\n",
      " [-0.19832905 -0.19832905 -0.19832905 ... -0.19861986 -0.19861986\n",
      "  -0.19861986]\n",
      " [-0.20237851 -0.20237851 -0.20237851 ... -0.21587732 -0.21587732\n",
      "  -0.21587732]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22544663 -0.22544663 -0.22544663 ... -0.1952735  -0.1952735\n",
      "  -0.1952735 ]\n",
      " [-0.20665851 -0.20665851 -0.20665851 ... -0.17428984 -0.17428984\n",
      "  -0.17428984]\n",
      " [-0.2013682  -0.2013682  -0.2013682  ... -0.2166346  -0.2166346\n",
      "  -0.2166346 ]\n",
      " ...\n",
      " [-0.18949059 -0.18949059 -0.18949059 ... -0.20727226 -0.20727226\n",
      "  -0.20727226]\n",
      " [-0.21654354 -0.21654354 -0.21654354 ... -0.22177017 -0.22177017\n",
      "  -0.22177017]\n",
      " [-0.21338445 -0.21338445 -0.21338445 ... -0.21841386 -0.21841386\n",
      "  -0.21841386]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.2043412  -0.2043412  -0.2043412  ... -0.22196153 -0.22196153\n",
      "  -0.22196153]\n",
      " [-0.20075016 -0.20075016 -0.20075016 ... -0.21154673 -0.21154673\n",
      "  -0.21154673]\n",
      " [-0.21634345 -0.21634345 -0.21634345 ... -0.21687451 -0.21687451\n",
      "  -0.21687451]\n",
      " ...\n",
      " [-0.20567372 -0.20567372 -0.20567372 ... -0.20517054 -0.20517054\n",
      "  -0.20517054]\n",
      " [-0.2167302  -0.2167302  -0.2167302  ... -0.20643297 -0.20643297\n",
      "  -0.20643297]\n",
      " [-0.21743433 -0.21743433 -0.21743433 ... -0.20673473 -0.20673473\n",
      "  -0.20673473]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.20187251 -0.20187251 -0.20187251 ... -0.21443959 -0.21443959\n",
      "  -0.21443959]\n",
      " [-0.21457532 -0.21457532 -0.21457532 ... -0.20930304 -0.20930304\n",
      "  -0.20930304]\n",
      " [-0.20300153 -0.20300153 -0.20300153 ... -0.21281557 -0.21281557\n",
      "  -0.21281557]\n",
      " ...\n",
      " [-0.23872247 -0.23872247 -0.23872247 ... -0.18714339 -0.18714339\n",
      "  -0.18714339]\n",
      " [-0.20357898 -0.20357898 -0.20357898 ... -0.23735993 -0.23735993\n",
      "  -0.23735993]\n",
      " [-0.21597788 -0.21597788 -0.21597788 ... -0.20832326 -0.20832326\n",
      "  -0.20832326]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.22065672 -0.22065672 -0.22065672 ... -0.21415918 -0.21415918\n",
      "  -0.21415918]\n",
      " [-0.2099447  -0.2099447  -0.2099447  ... -0.21109596 -0.21109596\n",
      "  -0.21109596]\n",
      " [-0.21811083 -0.21811083 -0.21811083 ... -0.21076907 -0.21076907\n",
      "  -0.21076907]\n",
      " ...\n",
      " [-0.23062292 -0.23062292 -0.23062292 ... -0.22287464 -0.22287464\n",
      "  -0.22287464]\n",
      " [-0.2130605  -0.2130605  -0.2130605  ... -0.21380097 -0.21380097\n",
      "  -0.21380097]\n",
      " [-0.21297869 -0.21297869 -0.21297869 ... -0.18727028 -0.18727028\n",
      "  -0.18727028]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.20305322 -0.20305322 -0.20305322 ... -0.24177697 -0.24177697\n",
      "  -0.24177697]\n",
      " [-0.16790843 -0.16790843 -0.16790843 ... -0.22708426 -0.22708426\n",
      "  -0.22708426]\n",
      " [-0.2130298  -0.2130298  -0.2130298  ... -0.20784162 -0.20784162\n",
      "  -0.20784162]\n",
      " ...\n",
      " [-0.21242051 -0.21242051 -0.21242051 ... -0.20085406 -0.20085406\n",
      "  -0.20085406]\n",
      " [-0.1980825  -0.1980825  -0.1980825  ... -0.21918303 -0.21918303\n",
      "  -0.21918303]\n",
      " [-0.2414239  -0.2414239  -0.2414239  ... -0.21954505 -0.21954505\n",
      "  -0.21954505]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  17 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23743829 -0.23743829 -0.23743829 ... -0.25772458 -0.25772458\n",
      "  -0.25772458]\n",
      " [-0.23551822 -0.23551822 -0.23551822 ... -0.24812412 -0.24812412\n",
      "  -0.24812412]\n",
      " [-0.2525234  -0.2525234  -0.2525234  ... -0.24318302 -0.24318302\n",
      "  -0.24318302]\n",
      " ...\n",
      " [-0.23711354 -0.23711354 -0.23711354 ... -0.23747781 -0.23747781\n",
      "  -0.23747781]\n",
      " [-0.23567633 -0.23567633 -0.23567633 ... -0.27313986 -0.27313986\n",
      "  -0.27313986]\n",
      " [-0.25305226 -0.25305226 -0.25305226 ... -0.22738689 -0.22738689\n",
      "  -0.22738689]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24132946 -0.24132946 -0.24132946 ... -0.25343946 -0.25343946\n",
      "  -0.25343946]\n",
      " [-0.24251632 -0.24251632 -0.24251632 ... -0.23870656 -0.23870656\n",
      "  -0.23870656]\n",
      " [-0.24655709 -0.24655709 -0.24655709 ... -0.24767105 -0.24767105\n",
      "  -0.24767105]\n",
      " ...\n",
      " [-0.2376337  -0.2376337  -0.2376337  ... -0.23230906 -0.23230906\n",
      "  -0.23230906]\n",
      " [-0.23713201 -0.23713201 -0.23713201 ... -0.2414644  -0.2414644\n",
      "  -0.2414644 ]\n",
      " [-0.24156138 -0.24156138 -0.24156138 ... -0.22728357 -0.22728357\n",
      "  -0.22728357]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23884141 -0.23884141 -0.23884141 ... -0.21920481 -0.21920481\n",
      "  -0.21920481]\n",
      " [-0.24050924 -0.24050924 -0.24050924 ... -0.27380335 -0.27380335\n",
      "  -0.27380335]\n",
      " [-0.23824947 -0.23824947 -0.23824947 ... -0.21019132 -0.21019132\n",
      "  -0.21019132]\n",
      " ...\n",
      " [-0.27090824 -0.27090824 -0.27090824 ... -0.25346112 -0.25346112\n",
      "  -0.25346112]\n",
      " [-0.24101707 -0.24101707 -0.24101707 ... -0.23619407 -0.23619407\n",
      "  -0.23619407]\n",
      " [-0.23186582 -0.23186582 -0.23186582 ... -0.23747781 -0.23747781\n",
      "  -0.23747781]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22848397 -0.22848397 -0.22848397 ... -0.23594913 -0.23594913\n",
      "  -0.23594913]\n",
      " [-0.2488042  -0.2488042  -0.2488042  ... -0.2306543  -0.2306543\n",
      "  -0.2306543 ]\n",
      " [-0.24275802 -0.24275802 -0.24275802 ... -0.24374396 -0.24374396\n",
      "  -0.24374396]\n",
      " ...\n",
      " [-0.23269475 -0.23269475 -0.23269475 ... -0.24988511 -0.24988511\n",
      "  -0.24988511]\n",
      " [-0.2517874  -0.2517874  -0.2517874  ... -0.26254255 -0.26254255\n",
      "  -0.26254255]\n",
      " [-0.26028126 -0.26028126 -0.26028126 ... -0.23918682 -0.23918682\n",
      "  -0.23918682]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.23421466 -0.23421466 -0.23421466 ... -0.2568261  -0.2568261\n",
      "  -0.2568261 ]\n",
      " [-0.24682224 -0.24682224 -0.24682224 ... -0.2693047  -0.2693047\n",
      "  -0.2693047 ]\n",
      " [-0.2507224  -0.2507224  -0.2507224  ... -0.26208046 -0.26208046\n",
      "  -0.26208046]\n",
      " ...\n",
      " [-0.26813853 -0.26813853 -0.26813853 ... -0.2287625  -0.2287625\n",
      "  -0.2287625 ]\n",
      " [-0.24010569 -0.24010569 -0.24010569 ... -0.23145233 -0.23145233\n",
      "  -0.23145233]\n",
      " [-0.24482794 -0.24482794 -0.24482794 ... -0.26235282 -0.26235282\n",
      "  -0.26235282]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.24646938 -0.24646938 -0.24646938 ... -0.25273222 -0.25273222\n",
      "  -0.25273222]\n",
      " [-0.24900225 -0.24900225 -0.24900225 ... -0.24287036 -0.24287036\n",
      "  -0.24287036]\n",
      " [-0.23952729 -0.23952729 -0.23952729 ... -0.24189693 -0.24189693\n",
      "  -0.24189693]\n",
      " ...\n",
      " [-0.25438547 -0.25438547 -0.25438547 ... -0.27080977 -0.27080977\n",
      "  -0.27080977]\n",
      " [-0.24990752 -0.24990752 -0.24990752 ... -0.2455307  -0.2455307\n",
      "  -0.2455307 ]\n",
      " [-0.2508093  -0.2508093  -0.2508093  ... -0.2786057  -0.2786057\n",
      "  -0.2786057 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24337555 -0.24337555 -0.24337555 ... -0.24975011 -0.24975011\n",
      "  -0.24975011]\n",
      " [-0.23456751 -0.23456751 -0.23456751 ... -0.26283932 -0.26283932\n",
      "  -0.26283932]\n",
      " [-0.2502813  -0.2502813  -0.2502813  ... -0.24424675 -0.24424675\n",
      "  -0.24424675]\n",
      " ...\n",
      " [-0.25351065 -0.25351065 -0.25351065 ... -0.23766625 -0.23766625\n",
      "  -0.23766625]\n",
      " [-0.27908373 -0.27908373 -0.27908373 ... -0.25129375 -0.25129375\n",
      "  -0.25129375]\n",
      " [-0.2453623  -0.2453623  -0.2453623  ... -0.24963848 -0.24963848\n",
      "  -0.24963848]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.24464926 -0.24464926 -0.24464926 ... -0.22898746 -0.22898746\n",
      "  -0.22898746]\n",
      " [-0.23882651 -0.23882651 -0.23882651 ... -0.24491434 -0.24491434\n",
      "  -0.24491434]\n",
      " [-0.24710476 -0.24710476 -0.24710476 ... -0.2350615  -0.2350615\n",
      "  -0.2350615 ]\n",
      " ...\n",
      " [-0.24147533 -0.24147533 -0.24147533 ... -0.2451283  -0.2451283\n",
      "  -0.2451283 ]\n",
      " [-0.21726494 -0.21726494 -0.21726494 ... -0.24496725 -0.24496725\n",
      "  -0.24496725]\n",
      " [-0.23561916 -0.23561916 -0.23561916 ... -0.2501064  -0.2501064\n",
      "  -0.2501064 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  18 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.28225255 -0.28225255 -0.28225255 ... -0.2652623  -0.2652623\n",
      "  -0.2652623 ]\n",
      " [-0.29595184 -0.29595184 -0.29595184 ... -0.24435014 -0.24435014\n",
      "  -0.24435014]\n",
      " [-0.2778905  -0.2778905  -0.2778905  ... -0.28574687 -0.28574687\n",
      "  -0.28574687]\n",
      " ...\n",
      " [-0.24807943 -0.24807943 -0.24807943 ... -0.27287102 -0.27287102\n",
      "  -0.27287102]\n",
      " [-0.26325783 -0.26325783 -0.26325783 ... -0.28209847 -0.28209847\n",
      "  -0.28209847]\n",
      " [-0.28699368 -0.28699368 -0.28699368 ... -0.26445645 -0.26445645\n",
      "  -0.26445645]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.25206006 -0.25206006 -0.25206006 ... -0.27569014 -0.27569014\n",
      "  -0.27569014]\n",
      " [-0.2672004  -0.2672004  -0.2672004  ... -0.23190486 -0.23190486\n",
      "  -0.23190486]\n",
      " [-0.25691646 -0.25691646 -0.25691646 ... -0.26783794 -0.26783794\n",
      "  -0.26783794]\n",
      " ...\n",
      " [-0.26268327 -0.26268327 -0.26268327 ... -0.27244312 -0.27244312\n",
      "  -0.27244312]\n",
      " [-0.2784934  -0.2784934  -0.2784934  ... -0.27900407 -0.27900407\n",
      "  -0.27900407]\n",
      " [-0.27251774 -0.27251774 -0.27251774 ... -0.2903614  -0.2903614\n",
      "  -0.2903614 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24996321 -0.24996321 -0.24996321 ... -0.28637636 -0.28637636\n",
      "  -0.28637636]\n",
      " [-0.2633215  -0.2633215  -0.2633215  ... -0.28249788 -0.28249788\n",
      "  -0.28249788]\n",
      " [-0.2753004  -0.2753004  -0.2753004  ... -0.27353173 -0.27353173\n",
      "  -0.27353173]\n",
      " ...\n",
      " [-0.27441263 -0.27441263 -0.27441263 ... -0.26616287 -0.26616287\n",
      "  -0.26616287]\n",
      " [-0.23016462 -0.23016462 -0.23016462 ... -0.25799218 -0.25799218\n",
      "  -0.25799218]\n",
      " [-0.27509034 -0.27509034 -0.27509034 ... -0.25135872 -0.25135872\n",
      "  -0.25135872]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2733365  -0.2733365  -0.2733365  ... -0.2742123  -0.2742123\n",
      "  -0.2742123 ]\n",
      " [-0.29739207 -0.29739207 -0.29739207 ... -0.27180433 -0.27180433\n",
      "  -0.27180433]\n",
      " [-0.27545646 -0.27545646 -0.27545646 ... -0.2750545  -0.2750545\n",
      "  -0.2750545 ]\n",
      " ...\n",
      " [-0.27336186 -0.27336186 -0.27336186 ... -0.3076688  -0.3076688\n",
      "  -0.3076688 ]\n",
      " [-0.27877948 -0.27877948 -0.27877948 ... -0.26681852 -0.26681852\n",
      "  -0.26681852]\n",
      " [-0.28380492 -0.28380492 -0.28380492 ... -0.26901442 -0.26901442\n",
      "  -0.26901442]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.2658773  -0.2658773  -0.2658773  ... -0.26306581 -0.26306581\n",
      "  -0.26306581]\n",
      " [-0.27186245 -0.27186245 -0.27186245 ... -0.25329784 -0.25329784\n",
      "  -0.25329784]\n",
      " [-0.26917553 -0.26917553 -0.26917553 ... -0.30759382 -0.30759382\n",
      "  -0.30759382]\n",
      " ...\n",
      " [-0.26161253 -0.26161253 -0.26161253 ... -0.25973052 -0.25973052\n",
      "  -0.25973052]\n",
      " [-0.2551477  -0.2551477  -0.2551477  ... -0.2799564  -0.2799564\n",
      "  -0.2799564 ]\n",
      " [-0.2881686  -0.2881686  -0.2881686  ... -0.27801067 -0.27801067\n",
      "  -0.27801067]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.28342938 -0.28342938 -0.28342938 ... -0.27436882 -0.27436882\n",
      "  -0.27436882]\n",
      " [-0.2855435  -0.2855435  -0.2855435  ... -0.29245213 -0.29245213\n",
      "  -0.29245213]\n",
      " [-0.2891578  -0.2891578  -0.2891578  ... -0.2986971  -0.2986971\n",
      "  -0.2986971 ]\n",
      " ...\n",
      " [-0.2756644  -0.2756644  -0.2756644  ... -0.27437592 -0.27437592\n",
      "  -0.27437592]\n",
      " [-0.27064425 -0.27064425 -0.27064425 ... -0.2774816  -0.2774816\n",
      "  -0.2774816 ]\n",
      " [-0.27680188 -0.27680188 -0.27680188 ... -0.2529573  -0.2529573\n",
      "  -0.2529573 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.27816853 -0.27816853 -0.27816853 ... -0.27224883 -0.27224883\n",
      "  -0.27224883]\n",
      " [-0.25756532 -0.25756532 -0.25756532 ... -0.28408715 -0.28408715\n",
      "  -0.28408715]\n",
      " [-0.29249313 -0.29249313 -0.29249313 ... -0.25599527 -0.25599527\n",
      "  -0.25599527]\n",
      " ...\n",
      " [-0.28378454 -0.28378454 -0.28378454 ... -0.25550324 -0.25550324\n",
      "  -0.25550324]\n",
      " [-0.26740682 -0.26740682 -0.26740682 ... -0.2755882  -0.2755882\n",
      "  -0.2755882 ]\n",
      " [-0.2728705  -0.2728705  -0.2728705  ... -0.26016486 -0.26016486\n",
      "  -0.26016486]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.25670066 -0.25670066 -0.25670066 ... -0.26480716 -0.26480716\n",
      "  -0.26480716]\n",
      " [-0.2996884  -0.2996884  -0.2996884  ... -0.2781372  -0.2781372\n",
      "  -0.2781372 ]\n",
      " [-0.2503391  -0.2503391  -0.2503391  ... -0.2795291  -0.2795291\n",
      "  -0.2795291 ]\n",
      " ...\n",
      " [-0.29183412 -0.29183412 -0.29183412 ... -0.29049268 -0.29049268\n",
      "  -0.29049268]\n",
      " [-0.27363747 -0.27363747 -0.27363747 ... -0.26074973 -0.26074973\n",
      "  -0.26074973]\n",
      " [-0.23404299 -0.23404299 -0.23404299 ... -0.27079996 -0.27079996\n",
      "  -0.27079996]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  19 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.27473578 -0.27473578 -0.27473578 ... -0.26348707 -0.26348707\n",
      "  -0.26348707]\n",
      " [-0.28045836 -0.28045836 -0.28045836 ... -0.2854694  -0.2854694\n",
      "  -0.2854694 ]\n",
      " [-0.28123003 -0.28123003 -0.28123003 ... -0.31627843 -0.31627843\n",
      "  -0.31627843]\n",
      " ...\n",
      " [-0.2808416  -0.2808416  -0.2808416  ... -0.31174248 -0.31174248\n",
      "  -0.31174248]\n",
      " [-0.27985775 -0.27985775 -0.27985775 ... -0.2651533  -0.2651533\n",
      "  -0.2651533 ]\n",
      " [-0.3027603  -0.3027603  -0.3027603  ... -0.29523116 -0.29523116\n",
      "  -0.29523116]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.28600985 -0.28600985 -0.28600985 ... -0.284774   -0.284774\n",
      "  -0.284774  ]\n",
      " [-0.28632465 -0.28632465 -0.28632465 ... -0.28094184 -0.28094184\n",
      "  -0.28094184]\n",
      " [-0.30381888 -0.30381888 -0.30381888 ... -0.29317588 -0.29317588\n",
      "  -0.29317588]\n",
      " ...\n",
      " [-0.28725857 -0.28725857 -0.28725857 ... -0.308411   -0.308411\n",
      "  -0.308411  ]\n",
      " [-0.30485877 -0.30485877 -0.30485877 ... -0.29271716 -0.29271716\n",
      "  -0.29271716]\n",
      " [-0.31265044 -0.31265044 -0.31265044 ... -0.2998377  -0.2998377\n",
      "  -0.2998377 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.26975578 -0.26975578 -0.26975578 ... -0.30223545 -0.30223545\n",
      "  -0.30223545]\n",
      " [-0.2915781  -0.2915781  -0.2915781  ... -0.2971279  -0.2971279\n",
      "  -0.2971279 ]\n",
      " [-0.2942743  -0.2942743  -0.2942743  ... -0.2914314  -0.2914314\n",
      "  -0.2914314 ]\n",
      " ...\n",
      " [-0.2992217  -0.2992217  -0.2992217  ... -0.28834748 -0.28834748\n",
      "  -0.28834748]\n",
      " [-0.32654426 -0.32654426 -0.32654426 ... -0.28077105 -0.28077105\n",
      "  -0.28077105]\n",
      " [-0.27173927 -0.27173927 -0.27173927 ... -0.28008926 -0.28008926\n",
      "  -0.28008926]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.292502   -0.292502   -0.292502   ... -0.29058772 -0.29058772\n",
      "  -0.29058772]\n",
      " [-0.29500753 -0.29500753 -0.29500753 ... -0.30033728 -0.30033728\n",
      "  -0.30033728]\n",
      " [-0.28260407 -0.28260407 -0.28260407 ... -0.30792144 -0.30792144\n",
      "  -0.30792144]\n",
      " ...\n",
      " [-0.2691943  -0.2691943  -0.2691943  ... -0.2886066  -0.2886066\n",
      "  -0.2886066 ]\n",
      " [-0.27094886 -0.27094886 -0.27094886 ... -0.3046472  -0.3046472\n",
      "  -0.3046472 ]\n",
      " [-0.26897866 -0.26897866 -0.26897866 ... -0.30293798 -0.30293798\n",
      "  -0.30293798]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.29803956 -0.29803956 -0.29803956 ... -0.2705652  -0.2705652\n",
      "  -0.2705652 ]\n",
      " [-0.2825922  -0.2825922  -0.2825922  ... -0.3071055  -0.3071055\n",
      "  -0.3071055 ]\n",
      " [-0.28773487 -0.28773487 -0.28773487 ... -0.28761134 -0.28761134\n",
      "  -0.28761134]\n",
      " ...\n",
      " [-0.3041297  -0.3041297  -0.3041297  ... -0.29763246 -0.29763246\n",
      "  -0.29763246]\n",
      " [-0.26378486 -0.26378486 -0.26378486 ... -0.30725634 -0.30725634\n",
      "  -0.30725634]\n",
      " [-0.3220865  -0.3220865  -0.3220865  ... -0.29153    -0.29153\n",
      "  -0.29153   ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.25301492 -0.25301492 -0.25301492 ... -0.27908328 -0.27908328\n",
      "  -0.27908328]\n",
      " [-0.29280233 -0.29280233 -0.29280233 ... -0.29239762 -0.29239762\n",
      "  -0.29239762]\n",
      " [-0.27323657 -0.27323657 -0.27323657 ... -0.28841612 -0.28841612\n",
      "  -0.28841612]\n",
      " ...\n",
      " [-0.2856898  -0.2856898  -0.2856898  ... -0.24818362 -0.24818362\n",
      "  -0.24818362]\n",
      " [-0.29727435 -0.29727435 -0.29727435 ... -0.2700888  -0.2700888\n",
      "  -0.2700888 ]\n",
      " [-0.24633406 -0.24633406 -0.24633406 ... -0.27692705 -0.27692705\n",
      "  -0.27692705]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.32459062 -0.32459062 -0.32459062 ... -0.29630098 -0.29630098\n",
      "  -0.29630098]\n",
      " [-0.27979195 -0.27979195 -0.27979195 ... -0.28446454 -0.28446454\n",
      "  -0.28446454]\n",
      " [-0.32120016 -0.32120016 -0.32120016 ... -0.3023486  -0.3023486\n",
      "  -0.3023486 ]\n",
      " ...\n",
      " [-0.2987324  -0.2987324  -0.2987324  ... -0.29954866 -0.29954866\n",
      "  -0.29954866]\n",
      " [-0.27587277 -0.27587277 -0.27587277 ... -0.2611919  -0.2611919\n",
      "  -0.2611919 ]\n",
      " [-0.29524815 -0.29524815 -0.29524815 ... -0.30175096 -0.30175096\n",
      "  -0.30175096]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.29633057 -0.29633057 -0.29633057 ... -0.2908244  -0.2908244\n",
      "  -0.2908244 ]\n",
      " [-0.29895183 -0.29895183 -0.29895183 ... -0.28547677 -0.28547677\n",
      "  -0.28547677]\n",
      " [-0.29106954 -0.29106954 -0.29106954 ... -0.27019548 -0.27019548\n",
      "  -0.27019548]\n",
      " ...\n",
      " [-0.31159544 -0.31159544 -0.31159544 ... -0.27967158 -0.27967158\n",
      "  -0.27967158]\n",
      " [-0.29898137 -0.29898137 -0.29898137 ... -0.29215658 -0.29215658\n",
      "  -0.29215658]\n",
      " [-0.29773998 -0.29773998 -0.29773998 ... -0.29082078 -0.29082078\n",
      "  -0.29082078]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  20 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.30255    -0.30255    -0.30255    ... -0.28497857 -0.28497857\n",
      "  -0.28497857]\n",
      " [-0.29824895 -0.29824895 -0.29824895 ... -0.3098632  -0.3098632\n",
      "  -0.3098632 ]\n",
      " [-0.29218602 -0.29218602 -0.29218602 ... -0.30056703 -0.30056703\n",
      "  -0.30056703]\n",
      " ...\n",
      " [-0.29464787 -0.29464787 -0.29464787 ... -0.2930225  -0.2930225\n",
      "  -0.2930225 ]\n",
      " [-0.31403974 -0.31403974 -0.31403974 ... -0.2974615  -0.2974615\n",
      "  -0.2974615 ]\n",
      " [-0.31429642 -0.31429642 -0.31429642 ... -0.3054897  -0.3054897\n",
      "  -0.3054897 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.35771415 -0.35771415 -0.35771415 ... -0.31378573 -0.31378573\n",
      "  -0.31378573]\n",
      " [-0.3228782  -0.3228782  -0.3228782  ... -0.31570673 -0.31570673\n",
      "  -0.31570673]\n",
      " [-0.2947835  -0.2947835  -0.2947835  ... -0.28730863 -0.28730863\n",
      "  -0.28730863]\n",
      " ...\n",
      " [-0.30530882 -0.30530882 -0.30530882 ... -0.29983908 -0.29983908\n",
      "  -0.29983908]\n",
      " [-0.29639393 -0.29639393 -0.29639393 ... -0.31859693 -0.31859693\n",
      "  -0.31859693]\n",
      " [-0.31331238 -0.31331238 -0.31331238 ... -0.31235495 -0.31235495\n",
      "  -0.31235495]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28595498 -0.28595498 -0.28595498 ... -0.310234   -0.310234\n",
      "  -0.310234  ]\n",
      " [-0.31607795 -0.31607795 -0.31607795 ... -0.30695608 -0.30695608\n",
      "  -0.30695608]\n",
      " [-0.2991314  -0.2991314  -0.2991314  ... -0.30557194 -0.30557194\n",
      "  -0.30557194]\n",
      " ...\n",
      " [-0.33769283 -0.33769283 -0.33769283 ... -0.27409035 -0.27409035\n",
      "  -0.27409035]\n",
      " [-0.29440135 -0.29440135 -0.29440135 ... -0.30095434 -0.30095434\n",
      "  -0.30095434]\n",
      " [-0.30197275 -0.30197275 -0.30197275 ... -0.29693192 -0.29693192\n",
      "  -0.29693192]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.272096   -0.272096   -0.272096   ... -0.2787896  -0.2787896\n",
      "  -0.2787896 ]\n",
      " [-0.30291173 -0.30291173 -0.30291173 ... -0.31020567 -0.31020567\n",
      "  -0.31020567]\n",
      " [-0.315825   -0.315825   -0.315825   ... -0.3209648  -0.3209648\n",
      "  -0.3209648 ]\n",
      " ...\n",
      " [-0.31495634 -0.31495634 -0.31495634 ... -0.30067375 -0.30067375\n",
      "  -0.30067375]\n",
      " [-0.28672132 -0.28672132 -0.28672132 ... -0.27884796 -0.27884796\n",
      "  -0.27884796]\n",
      " [-0.30862656 -0.30862656 -0.30862656 ... -0.3027483  -0.3027483\n",
      "  -0.3027483 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.29983652 -0.29983652 -0.29983652 ... -0.26162016 -0.26162016\n",
      "  -0.26162016]\n",
      " [-0.30362844 -0.30362844 -0.30362844 ... -0.34044814 -0.34044814\n",
      "  -0.34044814]\n",
      " [-0.32314736 -0.32314736 -0.32314736 ... -0.29525733 -0.29525733\n",
      "  -0.29525733]\n",
      " ...\n",
      " [-0.28631476 -0.28631476 -0.28631476 ... -0.32291484 -0.32291484\n",
      "  -0.32291484]\n",
      " [-0.29562888 -0.29562888 -0.29562888 ... -0.3087786  -0.3087786\n",
      "  -0.3087786 ]\n",
      " [-0.30301768 -0.30301768 -0.30301768 ... -0.31827885 -0.31827885\n",
      "  -0.31827885]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.31216872 -0.31216872 -0.31216872 ... -0.32067752 -0.32067752\n",
      "  -0.32067752]\n",
      " [-0.31083584 -0.31083584 -0.31083584 ... -0.35675436 -0.35675436\n",
      "  -0.35675436]\n",
      " [-0.29940504 -0.29940504 -0.29940504 ... -0.29724526 -0.29724526\n",
      "  -0.29724526]\n",
      " ...\n",
      " [-0.26818246 -0.26818246 -0.26818246 ... -0.31774655 -0.31774655\n",
      "  -0.31774655]\n",
      " [-0.30281448 -0.30281448 -0.30281448 ... -0.31599322 -0.31599322\n",
      "  -0.31599322]\n",
      " [-0.31013623 -0.31013623 -0.31013623 ... -0.3004937  -0.3004937\n",
      "  -0.3004937 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.30575544 -0.30575544 -0.30575544 ... -0.34124476 -0.34124476\n",
      "  -0.34124476]\n",
      " [-0.3130219  -0.3130219  -0.3130219  ... -0.3209648  -0.3209648\n",
      "  -0.3209648 ]\n",
      " [-0.31771305 -0.31771305 -0.31771305 ... -0.29035613 -0.29035613\n",
      "  -0.29035613]\n",
      " ...\n",
      " [-0.3079232  -0.3079232  -0.3079232  ... -0.31054032 -0.31054032\n",
      "  -0.31054032]\n",
      " [-0.30106276 -0.30106276 -0.30106276 ... -0.2402126  -0.2402126\n",
      "  -0.2402126 ]\n",
      " [-0.29014155 -0.29014155 -0.29014155 ... -0.32015476 -0.32015476\n",
      "  -0.32015476]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.3048494  -0.3048494  -0.3048494  ... -0.30029672 -0.30029672\n",
      "  -0.30029672]\n",
      " [-0.30054402 -0.30054402 -0.30054402 ... -0.35329476 -0.35329476\n",
      "  -0.35329476]\n",
      " [-0.31420445 -0.31420445 -0.31420445 ... -0.29422265 -0.29422265\n",
      "  -0.29422265]\n",
      " ...\n",
      " [-0.31812632 -0.31812632 -0.31812632 ... -0.28465566 -0.28465566\n",
      "  -0.28465566]\n",
      " [-0.31135207 -0.31135207 -0.31135207 ... -0.29506755 -0.29506755\n",
      "  -0.29506755]\n",
      " [-0.2912322  -0.2912322  -0.2912322  ... -0.3028943  -0.3028943\n",
      "  -0.3028943 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  21 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.31247804 -0.31247804 -0.31247804 ... -0.31547093 -0.31547093\n",
      "  -0.31547093]\n",
      " [-0.29827508 -0.29827508 -0.29827508 ... -0.32297772 -0.32297772\n",
      "  -0.32297772]\n",
      " [-0.3072104  -0.3072104  -0.3072104  ... -0.30313125 -0.30313125\n",
      "  -0.30313125]\n",
      " ...\n",
      " [-0.30402872 -0.30402872 -0.30402872 ... -0.3267989  -0.3267989\n",
      "  -0.3267989 ]\n",
      " [-0.3054996  -0.3054996  -0.3054996  ... -0.29826748 -0.29826748\n",
      "  -0.29826748]\n",
      " [-0.32928333 -0.32928333 -0.32928333 ... -0.322233   -0.322233\n",
      "  -0.322233  ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.29776847 -0.29776847 -0.29776847 ... -0.29817972 -0.29817972\n",
      "  -0.29817972]\n",
      " [-0.29545394 -0.29545394 -0.29545394 ... -0.31047332 -0.31047332\n",
      "  -0.31047332]\n",
      " [-0.29735374 -0.29735374 -0.29735374 ... -0.27213392 -0.27213392\n",
      "  -0.27213392]\n",
      " ...\n",
      " [-0.31217763 -0.31217763 -0.31217763 ... -0.30914095 -0.30914095\n",
      "  -0.30914095]\n",
      " [-0.2970324  -0.2970324  -0.2970324  ... -0.3158622  -0.3158622\n",
      "  -0.3158622 ]\n",
      " [-0.31458953 -0.31458953 -0.31458953 ... -0.32070115 -0.32070115\n",
      "  -0.32070115]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.29898986 -0.29898986 -0.29898986 ... -0.32224357 -0.32224357\n",
      "  -0.32224357]\n",
      " [-0.29065964 -0.29065964 -0.29065964 ... -0.29822767 -0.29822767\n",
      "  -0.29822767]\n",
      " [-0.3030774  -0.3030774  -0.3030774  ... -0.3318698  -0.3318698\n",
      "  -0.3318698 ]\n",
      " ...\n",
      " [-0.30829155 -0.30829155 -0.30829155 ... -0.32071772 -0.32071772\n",
      "  -0.32071772]\n",
      " [-0.31395417 -0.31395417 -0.31395417 ... -0.313829   -0.313829\n",
      "  -0.313829  ]\n",
      " [-0.30065888 -0.30065888 -0.30065888 ... -0.29159033 -0.29159033\n",
      "  -0.29159033]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.30709517 -0.30709517 -0.30709517 ... -0.3110794  -0.3110794\n",
      "  -0.3110794 ]\n",
      " [-0.32139677 -0.32139677 -0.32139677 ... -0.2874573  -0.2874573\n",
      "  -0.2874573 ]\n",
      " [-0.31838167 -0.31838167 -0.31838167 ... -0.29834858 -0.29834858\n",
      "  -0.29834858]\n",
      " ...\n",
      " [-0.31770325 -0.31770325 -0.31770325 ... -0.30237645 -0.30237645\n",
      "  -0.30237645]\n",
      " [-0.30633602 -0.30633602 -0.30633602 ... -0.32337338 -0.32337338\n",
      "  -0.32337338]\n",
      " [-0.3229974  -0.3229974  -0.3229974  ... -0.3096478  -0.3096478\n",
      "  -0.3096478 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.3130259  -0.3130259  -0.3130259  ... -0.30393782 -0.30393782\n",
      "  -0.30393782]\n",
      " [-0.30281705 -0.30281705 -0.30281705 ... -0.29278457 -0.29278457\n",
      "  -0.29278457]\n",
      " [-0.29702887 -0.29702887 -0.29702887 ... -0.3020603  -0.3020603\n",
      "  -0.3020603 ]\n",
      " ...\n",
      " [-0.32706735 -0.32706735 -0.32706735 ... -0.3026916  -0.3026916\n",
      "  -0.3026916 ]\n",
      " [-0.30187917 -0.30187917 -0.30187917 ... -0.31205237 -0.31205237\n",
      "  -0.31205237]\n",
      " [-0.30834222 -0.30834222 -0.30834222 ... -0.29553258 -0.29553258\n",
      "  -0.29553258]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.29065168 -0.29065168 -0.29065168 ... -0.29075843 -0.29075843\n",
      "  -0.29075843]\n",
      " [-0.29850614 -0.29850614 -0.29850614 ... -0.3150673  -0.3150673\n",
      "  -0.3150673 ]\n",
      " [-0.3013112  -0.3013112  -0.3013112  ... -0.33522725 -0.33522725\n",
      "  -0.33522725]\n",
      " ...\n",
      " [-0.31917706 -0.31917706 -0.31917706 ... -0.31073958 -0.31073958\n",
      "  -0.31073958]\n",
      " [-0.29861796 -0.29861796 -0.29861796 ... -0.2946137  -0.2946137\n",
      "  -0.2946137 ]\n",
      " [-0.2992937  -0.2992937  -0.2992937  ... -0.29944366 -0.29944366\n",
      "  -0.29944366]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.29745096 -0.29745096 -0.29745096 ... -0.31823003 -0.31823003\n",
      "  -0.31823003]\n",
      " [-0.3004898  -0.3004898  -0.3004898  ... -0.3247779  -0.3247779\n",
      "  -0.3247779 ]\n",
      " [-0.30003297 -0.30003297 -0.30003297 ... -0.28828368 -0.28828368\n",
      "  -0.28828368]\n",
      " ...\n",
      " [-0.31902593 -0.31902593 -0.31902593 ... -0.32090008 -0.32090008\n",
      "  -0.32090008]\n",
      " [-0.32977095 -0.32977095 -0.32977095 ... -0.3313347  -0.3313347\n",
      "  -0.3313347 ]\n",
      " [-0.30070722 -0.30070722 -0.30070722 ... -0.31342465 -0.31342465\n",
      "  -0.31342465]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.31244856 -0.31244856 -0.31244856 ... -0.34860262 -0.34860262\n",
      "  -0.34860262]\n",
      " [-0.30840015 -0.30840015 -0.30840015 ... -0.3188494  -0.3188494\n",
      "  -0.3188494 ]\n",
      " [-0.3257746  -0.3257746  -0.3257746  ... -0.33041015 -0.33041015\n",
      "  -0.33041015]\n",
      " ...\n",
      " [-0.329138   -0.329138   -0.329138   ... -0.3041086  -0.3041086\n",
      "  -0.3041086 ]\n",
      " [-0.2913352  -0.2913352  -0.2913352  ... -0.2952156  -0.2952156\n",
      "  -0.2952156 ]\n",
      " [-0.31712857 -0.31712857 -0.31712857 ... -0.28925794 -0.28925794\n",
      "  -0.28925794]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  22 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.3002702  -0.3002702  -0.3002702  ... -0.32113218 -0.32113218\n",
      "  -0.32113218]\n",
      " [-0.31445616 -0.31445616 -0.31445616 ... -0.31821615 -0.31821615\n",
      "  -0.31821615]\n",
      " [-0.28149    -0.28149    -0.28149    ... -0.3089064  -0.3089064\n",
      "  -0.3089064 ]\n",
      " ...\n",
      " [-0.2978853  -0.2978853  -0.2978853  ... -0.28884187 -0.28884187\n",
      "  -0.28884187]\n",
      " [-0.3159441  -0.3159441  -0.3159441  ... -0.299038   -0.299038\n",
      "  -0.299038  ]\n",
      " [-0.31383234 -0.31383234 -0.31383234 ... -0.28291753 -0.28291753\n",
      "  -0.28291753]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.30163795 -0.30163795 -0.30163795 ... -0.30443287 -0.30443287\n",
      "  -0.30443287]\n",
      " [-0.30127174 -0.30127174 -0.30127174 ... -0.29014465 -0.29014465\n",
      "  -0.29014465]\n",
      " [-0.30801424 -0.30801424 -0.30801424 ... -0.3141367  -0.3141367\n",
      "  -0.3141367 ]\n",
      " ...\n",
      " [-0.31994525 -0.31994525 -0.31994525 ... -0.30602917 -0.30602917\n",
      "  -0.30602917]\n",
      " [-0.32098782 -0.32098782 -0.32098782 ... -0.28889057 -0.28889057\n",
      "  -0.28889057]\n",
      " [-0.31742522 -0.31742522 -0.31742522 ... -0.30598035 -0.30598035\n",
      "  -0.30598035]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.29574737 -0.29574737 -0.29574737 ... -0.31382644 -0.31382644\n",
      "  -0.31382644]\n",
      " [-0.30312836 -0.30312836 -0.30312836 ... -0.29579574 -0.29579574\n",
      "  -0.29579574]\n",
      " [-0.3135979  -0.3135979  -0.3135979  ... -0.30405468 -0.30405468\n",
      "  -0.30405468]\n",
      " ...\n",
      " [-0.2880858  -0.2880858  -0.2880858  ... -0.30644953 -0.30644953\n",
      "  -0.30644953]\n",
      " [-0.29721415 -0.29721415 -0.29721415 ... -0.29520255 -0.29520255\n",
      "  -0.29520255]\n",
      " [-0.3021372  -0.3021372  -0.3021372  ... -0.33454043 -0.33454043\n",
      "  -0.33454043]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.28618854 -0.28618854 -0.28618854 ... -0.30791402 -0.30791402\n",
      "  -0.30791402]\n",
      " [-0.31374872 -0.31374872 -0.31374872 ... -0.30580887 -0.30580887\n",
      "  -0.30580887]\n",
      " [-0.29540712 -0.29540712 -0.29540712 ... -0.30889642 -0.30889642\n",
      "  -0.30889642]\n",
      " ...\n",
      " [-0.3154093  -0.3154093  -0.3154093  ... -0.29786956 -0.29786956\n",
      "  -0.29786956]\n",
      " [-0.25887832 -0.25887832 -0.25887832 ... -0.30148754 -0.30148754\n",
      "  -0.30148754]\n",
      " [-0.28065595 -0.28065595 -0.28065595 ... -0.31115043 -0.31115043\n",
      "  -0.31115043]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.29005265 -0.29005265 -0.29005265 ... -0.32030776 -0.32030776\n",
      "  -0.32030776]\n",
      " [-0.29547474 -0.29547474 -0.29547474 ... -0.31358996 -0.31358996\n",
      "  -0.31358996]\n",
      " [-0.30352014 -0.30352014 -0.30352014 ... -0.2898853  -0.2898853\n",
      "  -0.2898853 ]\n",
      " ...\n",
      " [-0.2858336  -0.2858336  -0.2858336  ... -0.3261795  -0.3261795\n",
      "  -0.3261795 ]\n",
      " [-0.31465206 -0.31465206 -0.31465206 ... -0.29653272 -0.29653272\n",
      "  -0.29653272]\n",
      " [-0.26832008 -0.26832008 -0.26832008 ... -0.2857337  -0.2857337\n",
      "  -0.2857337 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.30389476 -0.30389476 -0.30389476 ... -0.311377   -0.311377\n",
      "  -0.311377  ]\n",
      " [-0.31641287 -0.31641287 -0.31641287 ... -0.33570024 -0.33570024\n",
      "  -0.33570024]\n",
      " [-0.3212894  -0.3212894  -0.3212894  ... -0.316454   -0.316454\n",
      "  -0.316454  ]\n",
      " ...\n",
      " [-0.3195336  -0.3195336  -0.3195336  ... -0.2801398  -0.2801398\n",
      "  -0.2801398 ]\n",
      " [-0.26625675 -0.26625675 -0.26625675 ... -0.30559725 -0.30559725\n",
      "  -0.30559725]\n",
      " [-0.29793125 -0.29793125 -0.29793125 ... -0.2771088  -0.2771088\n",
      "  -0.2771088 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.29299772 -0.29299772 -0.29299772 ... -0.3225694  -0.3225694\n",
      "  -0.3225694 ]\n",
      " [-0.2655877  -0.2655877  -0.2655877  ... -0.31157383 -0.31157383\n",
      "  -0.31157383]\n",
      " [-0.31121832 -0.31121832 -0.31121832 ... -0.31401783 -0.31401783\n",
      "  -0.31401783]\n",
      " ...\n",
      " [-0.29890364 -0.29890364 -0.29890364 ... -0.3101189  -0.3101189\n",
      "  -0.3101189 ]\n",
      " [-0.2891509  -0.2891509  -0.2891509  ... -0.30635542 -0.30635542\n",
      "  -0.30635542]\n",
      " [-0.31864154 -0.31864154 -0.31864154 ... -0.3063575  -0.3063575\n",
      "  -0.3063575 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.3126345  -0.3126345  -0.3126345  ... -0.28570563 -0.28570563\n",
      "  -0.28570563]\n",
      " [-0.33316174 -0.33316174 -0.33316174 ... -0.32011336 -0.32011336\n",
      "  -0.32011336]\n",
      " [-0.2780783  -0.2780783  -0.2780783  ... -0.30434728 -0.30434728\n",
      "  -0.30434728]\n",
      " ...\n",
      " [-0.29923806 -0.29923806 -0.29923806 ... -0.29945618 -0.29945618\n",
      "  -0.29945618]\n",
      " [-0.33087528 -0.33087528 -0.33087528 ... -0.2901312  -0.2901312\n",
      "  -0.2901312 ]\n",
      " [-0.29900724 -0.29900724 -0.29900724 ... -0.30842775 -0.30842775\n",
      "  -0.30842775]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  23 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2957878  -0.2957878  -0.2957878  ... -0.2963304  -0.2963304\n",
      "  -0.2963304 ]\n",
      " [-0.29360607 -0.29360607 -0.29360607 ... -0.26991126 -0.26991126\n",
      "  -0.26991126]\n",
      " [-0.26899207 -0.26899207 -0.26899207 ... -0.29569364 -0.29569364\n",
      "  -0.29569364]\n",
      " ...\n",
      " [-0.33666846 -0.33666846 -0.33666846 ... -0.273237   -0.273237\n",
      "  -0.273237  ]\n",
      " [-0.29354313 -0.29354313 -0.29354313 ... -0.28781754 -0.28781754\n",
      "  -0.28781754]\n",
      " [-0.29791945 -0.29791945 -0.29791945 ... -0.3378684  -0.3378684\n",
      "  -0.3378684 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.29860076 -0.29860076 -0.29860076 ... -0.27127138 -0.27127138\n",
      "  -0.27127138]\n",
      " [-0.2812073  -0.2812073  -0.2812073  ... -0.2973976  -0.2973976\n",
      "  -0.2973976 ]\n",
      " [-0.32004237 -0.32004237 -0.32004237 ... -0.30378863 -0.30378863\n",
      "  -0.30378863]\n",
      " ...\n",
      " [-0.287492   -0.287492   -0.287492   ... -0.29459187 -0.29459187\n",
      "  -0.29459187]\n",
      " [-0.287811   -0.287811   -0.287811   ... -0.30816954 -0.30816954\n",
      "  -0.30816954]\n",
      " [-0.27115905 -0.27115905 -0.27115905 ... -0.2920748  -0.2920748\n",
      "  -0.2920748 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.32679802 -0.32679802 -0.32679802 ... -0.28490466 -0.28490466\n",
      "  -0.28490466]\n",
      " [-0.32277796 -0.32277796 -0.32277796 ... -0.28519845 -0.28519845\n",
      "  -0.28519845]\n",
      " [-0.29003224 -0.29003224 -0.29003224 ... -0.302391   -0.302391\n",
      "  -0.302391  ]\n",
      " ...\n",
      " [-0.30676615 -0.30676615 -0.30676615 ... -0.26546696 -0.26546696\n",
      "  -0.26546696]\n",
      " [-0.3117358  -0.3117358  -0.3117358  ... -0.30610785 -0.30610785\n",
      "  -0.30610785]\n",
      " [-0.3060902  -0.3060902  -0.3060902  ... -0.28750312 -0.28750312\n",
      "  -0.28750312]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.25792086 -0.25792086 -0.25792086 ... -0.2931379  -0.2931379\n",
      "  -0.2931379 ]\n",
      " [-0.2634561  -0.2634561  -0.2634561  ... -0.28878707 -0.28878707\n",
      "  -0.28878707]\n",
      " [-0.2774701  -0.2774701  -0.2774701  ... -0.3065436  -0.3065436\n",
      "  -0.3065436 ]\n",
      " ...\n",
      " [-0.30508286 -0.30508286 -0.30508286 ... -0.2860775  -0.2860775\n",
      "  -0.2860775 ]\n",
      " [-0.28921056 -0.28921056 -0.28921056 ... -0.2910802  -0.2910802\n",
      "  -0.2910802 ]\n",
      " [-0.30547923 -0.30547923 -0.30547923 ... -0.30454433 -0.30454433\n",
      "  -0.30454433]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.30377775 -0.30377775 -0.30377775 ... -0.30955228 -0.30955228\n",
      "  -0.30955228]\n",
      " [-0.30483183 -0.30483183 -0.30483183 ... -0.321759   -0.321759\n",
      "  -0.321759  ]\n",
      " [-0.28138453 -0.28138453 -0.28138453 ... -0.3023766  -0.3023766\n",
      "  -0.3023766 ]\n",
      " ...\n",
      " [-0.27702296 -0.27702296 -0.27702296 ... -0.29453996 -0.29453996\n",
      "  -0.29453996]\n",
      " [-0.3057864  -0.3057864  -0.3057864  ... -0.28622884 -0.28622884\n",
      "  -0.28622884]\n",
      " [-0.28357768 -0.28357768 -0.28357768 ... -0.26267898 -0.26267898\n",
      "  -0.26267898]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.3156618  -0.3156618  -0.3156618  ... -0.30064082 -0.30064082\n",
      "  -0.30064082]\n",
      " [-0.30963588 -0.30963588 -0.30963588 ... -0.2816664  -0.2816664\n",
      "  -0.2816664 ]\n",
      " [-0.30529994 -0.30529994 -0.30529994 ... -0.2728112  -0.2728112\n",
      "  -0.2728112 ]\n",
      " ...\n",
      " [-0.30409133 -0.30409133 -0.30409133 ... -0.29545635 -0.29545635\n",
      "  -0.29545635]\n",
      " [-0.29973716 -0.29973716 -0.29973716 ... -0.30636021 -0.30636021\n",
      "  -0.30636021]\n",
      " [-0.33174628 -0.33174628 -0.33174628 ... -0.3006918  -0.3006918\n",
      "  -0.3006918 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.30320913 -0.30320913 -0.30320913 ... -0.27013162 -0.27013162\n",
      "  -0.27013162]\n",
      " [-0.30777925 -0.30777925 -0.30777925 ... -0.31284818 -0.31284818\n",
      "  -0.31284818]\n",
      " [-0.2722408  -0.2722408  -0.2722408  ... -0.29853046 -0.29853046\n",
      "  -0.29853046]\n",
      " ...\n",
      " [-0.31085694 -0.31085694 -0.31085694 ... -0.2856942  -0.2856942\n",
      "  -0.2856942 ]\n",
      " [-0.29838222 -0.29838222 -0.29838222 ... -0.27852172 -0.27852172\n",
      "  -0.27852172]\n",
      " [-0.28285748 -0.28285748 -0.28285748 ... -0.28823406 -0.28823406\n",
      "  -0.28823406]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.27822164 -0.27822164 -0.27822164 ... -0.35190058 -0.35190058\n",
      "  -0.35190058]\n",
      " [-0.30636114 -0.30636114 -0.30636114 ... -0.28472117 -0.28472117\n",
      "  -0.28472117]\n",
      " [-0.31666684 -0.31666684 -0.31666684 ... -0.28041637 -0.28041637\n",
      "  -0.28041637]\n",
      " ...\n",
      " [-0.28067324 -0.28067324 -0.28067324 ... -0.29079267 -0.29079267\n",
      "  -0.29079267]\n",
      " [-0.32031    -0.32031    -0.32031    ... -0.2895241  -0.2895241\n",
      "  -0.2895241 ]\n",
      " [-0.29284298 -0.29284298 -0.29284298 ... -0.30878788 -0.30878788\n",
      "  -0.30878788]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  24 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.26861715 -0.26861715 -0.26861715 ... -0.2935297  -0.2935297\n",
      "  -0.2935297 ]\n",
      " [-0.27740425 -0.27740425 -0.27740425 ... -0.257448   -0.257448\n",
      "  -0.257448  ]\n",
      " [-0.2925169  -0.2925169  -0.2925169  ... -0.30540457 -0.30540457\n",
      "  -0.30540457]\n",
      " ...\n",
      " [-0.2813642  -0.2813642  -0.2813642  ... -0.27291226 -0.27291226\n",
      "  -0.27291226]\n",
      " [-0.2631308  -0.2631308  -0.2631308  ... -0.29082248 -0.29082248\n",
      "  -0.29082248]\n",
      " [-0.25941348 -0.25941348 -0.25941348 ... -0.2675142  -0.2675142\n",
      "  -0.2675142 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.27076694 -0.27076694 -0.27076694 ... -0.28881466 -0.28881466\n",
      "  -0.28881466]\n",
      " [-0.26801455 -0.26801455 -0.26801455 ... -0.273818   -0.273818\n",
      "  -0.273818  ]\n",
      " [-0.29190683 -0.29190683 -0.29190683 ... -0.25628245 -0.25628245\n",
      "  -0.25628245]\n",
      " ...\n",
      " [-0.27664906 -0.27664906 -0.27664906 ... -0.27869886 -0.27869886\n",
      "  -0.27869886]\n",
      " [-0.2865594  -0.2865594  -0.2865594  ... -0.25671777 -0.25671777\n",
      "  -0.25671777]\n",
      " [-0.2650357  -0.2650357  -0.2650357  ... -0.29027477 -0.29027477\n",
      "  -0.29027477]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.29685238 -0.29685238 -0.29685238 ... -0.3012647  -0.3012647\n",
      "  -0.3012647 ]\n",
      " [-0.2691505  -0.2691505  -0.2691505  ... -0.27042475 -0.27042475\n",
      "  -0.27042475]\n",
      " [-0.28787333 -0.28787333 -0.28787333 ... -0.29602796 -0.29602796\n",
      "  -0.29602796]\n",
      " ...\n",
      " [-0.2736688  -0.2736688  -0.2736688  ... -0.29567957 -0.29567957\n",
      "  -0.29567957]\n",
      " [-0.28896016 -0.28896016 -0.28896016 ... -0.28717542 -0.28717542\n",
      "  -0.28717542]\n",
      " [-0.3169282  -0.3169282  -0.3169282  ... -0.28581458 -0.28581458\n",
      "  -0.28581458]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.3047988  -0.3047988  -0.3047988  ... -0.2789765  -0.2789765\n",
      "  -0.2789765 ]\n",
      " [-0.26978713 -0.26978713 -0.26978713 ... -0.27339226 -0.27339226\n",
      "  -0.27339226]\n",
      " [-0.2966706  -0.2966706  -0.2966706  ... -0.2670429  -0.2670429\n",
      "  -0.2670429 ]\n",
      " ...\n",
      " [-0.28191575 -0.28191575 -0.28191575 ... -0.2714569  -0.2714569\n",
      "  -0.2714569 ]\n",
      " [-0.2809286  -0.2809286  -0.2809286  ... -0.2634021  -0.2634021\n",
      "  -0.2634021 ]\n",
      " [-0.31621772 -0.31621772 -0.31621772 ... -0.27140945 -0.27140945\n",
      "  -0.27140945]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.29515415 -0.29515415 -0.29515415 ... -0.26739794 -0.26739794\n",
      "  -0.26739794]\n",
      " [-0.26111662 -0.26111662 -0.26111662 ... -0.28745744 -0.28745744\n",
      "  -0.28745744]\n",
      " [-0.26558715 -0.26558715 -0.26558715 ... -0.29245594 -0.29245594\n",
      "  -0.29245594]\n",
      " ...\n",
      " [-0.30239975 -0.30239975 -0.30239975 ... -0.25941086 -0.25941086\n",
      "  -0.25941086]\n",
      " [-0.26413634 -0.26413634 -0.26413634 ... -0.29521796 -0.29521796\n",
      "  -0.29521796]\n",
      " [-0.29142538 -0.29142538 -0.29142538 ... -0.28180027 -0.28180027\n",
      "  -0.28180027]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.3204     -0.3204     -0.3204     ... -0.29777247 -0.29777247\n",
      "  -0.29777247]\n",
      " [-0.2911575  -0.2911575  -0.2911575  ... -0.27798915 -0.27798915\n",
      "  -0.27798915]\n",
      " [-0.2862773  -0.2862773  -0.2862773  ... -0.2876306  -0.2876306\n",
      "  -0.2876306 ]\n",
      " ...\n",
      " [-0.269787   -0.269787   -0.269787   ... -0.28930783 -0.28930783\n",
      "  -0.28930783]\n",
      " [-0.26864362 -0.26864362 -0.26864362 ... -0.28585416 -0.28585416\n",
      "  -0.28585416]\n",
      " [-0.29797328 -0.29797328 -0.29797328 ... -0.3011763  -0.3011763\n",
      "  -0.3011763 ]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.30053863 -0.30053863 -0.30053863 ... -0.27588814 -0.27588814\n",
      "  -0.27588814]\n",
      " [-0.29141444 -0.29141444 -0.29141444 ... -0.27412316 -0.27412316\n",
      "  -0.27412316]\n",
      " [-0.2868231  -0.2868231  -0.2868231  ... -0.27037624 -0.27037624\n",
      "  -0.27037624]\n",
      " ...\n",
      " [-0.26999637 -0.26999637 -0.26999637 ... -0.25484666 -0.25484666\n",
      "  -0.25484666]\n",
      " [-0.2597258  -0.2597258  -0.2597258  ... -0.27153245 -0.27153245\n",
      "  -0.27153245]\n",
      " [-0.26966798 -0.26966798 -0.26966798 ... -0.26386246 -0.26386246\n",
      "  -0.26386246]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.25391847 -0.25391847 -0.25391847 ... -0.27450976 -0.27450976\n",
      "  -0.27450976]\n",
      " [-0.27557182 -0.27557182 -0.27557182 ... -0.2853668  -0.2853668\n",
      "  -0.2853668 ]\n",
      " [-0.27411473 -0.27411473 -0.27411473 ... -0.25851893 -0.25851893\n",
      "  -0.25851893]\n",
      " ...\n",
      " [-0.27787444 -0.27787444 -0.27787444 ... -0.27529323 -0.27529323\n",
      "  -0.27529323]\n",
      " [-0.27665532 -0.27665532 -0.27665532 ... -0.27657032 -0.27657032\n",
      "  -0.27657032]\n",
      " [-0.26452506 -0.26452506 -0.26452506 ... -0.25721875 -0.25721875\n",
      "  -0.25721875]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  25 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.27114213 -0.27114213 -0.27114213 ... -0.25259358 -0.25259358\n",
      "  -0.25259358]\n",
      " [-0.26357958 -0.26357958 -0.26357958 ... -0.24389115 -0.24389115\n",
      "  -0.24389115]\n",
      " [-0.27596626 -0.27596626 -0.27596626 ... -0.24856132 -0.24856132\n",
      "  -0.24856132]\n",
      " ...\n",
      " [-0.26351926 -0.26351926 -0.26351926 ... -0.23031056 -0.23031056\n",
      "  -0.23031056]\n",
      " [-0.27249363 -0.27249363 -0.27249363 ... -0.26139614 -0.26139614\n",
      "  -0.26139614]\n",
      " [-0.27155653 -0.27155653 -0.27155653 ... -0.26349425 -0.26349425\n",
      "  -0.26349425]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.26974338 -0.26974338 -0.26974338 ... -0.23676646 -0.23676646\n",
      "  -0.23676646]\n",
      " [-0.2572597  -0.2572597  -0.2572597  ... -0.27335638 -0.27335638\n",
      "  -0.27335638]\n",
      " [-0.2509351  -0.2509351  -0.2509351  ... -0.25318936 -0.25318936\n",
      "  -0.25318936]\n",
      " ...\n",
      " [-0.26417425 -0.26417425 -0.26417425 ... -0.27388006 -0.27388006\n",
      "  -0.27388006]\n",
      " [-0.27422374 -0.27422374 -0.27422374 ... -0.21577004 -0.21577004\n",
      "  -0.21577004]\n",
      " [-0.2654236  -0.2654236  -0.2654236  ... -0.234983   -0.234983\n",
      "  -0.234983  ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28559148 -0.28559148 -0.28559148 ... -0.25412425 -0.25412425\n",
      "  -0.25412425]\n",
      " [-0.27318358 -0.27318358 -0.27318358 ... -0.25809556 -0.25809556\n",
      "  -0.25809556]\n",
      " [-0.27256992 -0.27256992 -0.27256992 ... -0.26072568 -0.26072568\n",
      "  -0.26072568]\n",
      " ...\n",
      " [-0.25825983 -0.25825983 -0.25825983 ... -0.25921637 -0.25921637\n",
      "  -0.25921637]\n",
      " [-0.26266047 -0.26266047 -0.26266047 ... -0.24842119 -0.24842119\n",
      "  -0.24842119]\n",
      " [-0.278349   -0.278349   -0.278349   ... -0.2470245  -0.2470245\n",
      "  -0.2470245 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.26356706 -0.26356706 -0.26356706 ... -0.26289552 -0.26289552\n",
      "  -0.26289552]\n",
      " [-0.26311857 -0.26311857 -0.26311857 ... -0.25175428 -0.25175428\n",
      "  -0.25175428]\n",
      " [-0.26437783 -0.26437783 -0.26437783 ... -0.23971216 -0.23971216\n",
      "  -0.23971216]\n",
      " ...\n",
      " [-0.27134916 -0.27134916 -0.27134916 ... -0.26634285 -0.26634285\n",
      "  -0.26634285]\n",
      " [-0.2642144  -0.2642144  -0.2642144  ... -0.26986283 -0.26986283\n",
      "  -0.26986283]\n",
      " [-0.25701573 -0.25701573 -0.25701573 ... -0.25398487 -0.25398487\n",
      "  -0.25398487]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.24832766 -0.24832766 -0.24832766 ... -0.25935304 -0.25935304\n",
      "  -0.25935304]\n",
      " [-0.27070978 -0.27070978 -0.27070978 ... -0.28519043 -0.28519043\n",
      "  -0.28519043]\n",
      " [-0.2591358  -0.2591358  -0.2591358  ... -0.30630854 -0.30630854\n",
      "  -0.30630854]\n",
      " ...\n",
      " [-0.28282028 -0.28282028 -0.28282028 ... -0.26611456 -0.26611456\n",
      "  -0.26611456]\n",
      " [-0.2650744  -0.2650744  -0.2650744  ... -0.2831441  -0.2831441\n",
      "  -0.2831441 ]\n",
      " [-0.27144936 -0.27144936 -0.27144936 ... -0.26069224 -0.26069224\n",
      "  -0.26069224]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.24911925 -0.24911925 -0.24911925 ... -0.27132162 -0.27132162\n",
      "  -0.27132162]\n",
      " [-0.280697   -0.280697   -0.280697   ... -0.24924889 -0.24924889\n",
      "  -0.24924889]\n",
      " [-0.24328369 -0.24328369 -0.24328369 ... -0.27375075 -0.27375075\n",
      "  -0.27375075]\n",
      " ...\n",
      " [-0.25287604 -0.25287604 -0.25287604 ... -0.2694881  -0.2694881\n",
      "  -0.2694881 ]\n",
      " [-0.25984055 -0.25984055 -0.25984055 ... -0.26064223 -0.26064223\n",
      "  -0.26064223]\n",
      " [-0.2356993  -0.2356993  -0.2356993  ... -0.27989373 -0.27989373\n",
      "  -0.27989373]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24091405 -0.24091405 -0.24091405 ... -0.23918165 -0.23918165\n",
      "  -0.23918165]\n",
      " [-0.27149805 -0.27149805 -0.27149805 ... -0.26159948 -0.26159948\n",
      "  -0.26159948]\n",
      " [-0.2531241  -0.2531241  -0.2531241  ... -0.28206736 -0.28206736\n",
      "  -0.28206736]\n",
      " ...\n",
      " [-0.2571476  -0.2571476  -0.2571476  ... -0.24135791 -0.24135791\n",
      "  -0.24135791]\n",
      " [-0.2607181  -0.2607181  -0.2607181  ... -0.2656159  -0.2656159\n",
      "  -0.2656159 ]\n",
      " [-0.2621504  -0.2621504  -0.2621504  ... -0.2820073  -0.2820073\n",
      "  -0.2820073 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.27932608 -0.27932608 -0.27932608 ... -0.26404962 -0.26404962\n",
      "  -0.26404962]\n",
      " [-0.258158   -0.258158   -0.258158   ... -0.2402407  -0.2402407\n",
      "  -0.2402407 ]\n",
      " [-0.27116352 -0.27116352 -0.27116352 ... -0.2753255  -0.2753255\n",
      "  -0.2753255 ]\n",
      " ...\n",
      " [-0.29495057 -0.29495057 -0.29495057 ... -0.24477512 -0.24477512\n",
      "  -0.24477512]\n",
      " [-0.23777506 -0.23777506 -0.23777506 ... -0.27463365 -0.27463365\n",
      "  -0.27463365]\n",
      " [-0.27356717 -0.27356717 -0.27356717 ... -0.28042933 -0.28042933\n",
      "  -0.28042933]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  26 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.21730551 -0.21730551 -0.21730551 ... -0.23798622 -0.23798622\n",
      "  -0.23798622]\n",
      " [-0.19460799 -0.19460799 -0.19460799 ... -0.24834624 -0.24834624\n",
      "  -0.24834624]\n",
      " [-0.2510563  -0.2510563  -0.2510563  ... -0.22299616 -0.22299616\n",
      "  -0.22299616]\n",
      " ...\n",
      " [-0.26554498 -0.26554498 -0.26554498 ... -0.21597014 -0.21597014\n",
      "  -0.21597014]\n",
      " [-0.24453472 -0.24453472 -0.24453472 ... -0.25932938 -0.25932938\n",
      "  -0.25932938]\n",
      " [-0.2257804  -0.2257804  -0.2257804  ... -0.24375796 -0.24375796\n",
      "  -0.24375796]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.28554806 -0.28554806 -0.28554806 ... -0.24119747 -0.24119747\n",
      "  -0.24119747]\n",
      " [-0.22624932 -0.22624932 -0.22624932 ... -0.21133211 -0.21133211\n",
      "  -0.21133211]\n",
      " [-0.23551771 -0.23551771 -0.23551771 ... -0.22453736 -0.22453736\n",
      "  -0.22453736]\n",
      " ...\n",
      " [-0.235907   -0.235907   -0.235907   ... -0.24049407 -0.24049407\n",
      "  -0.24049407]\n",
      " [-0.23508334 -0.23508334 -0.23508334 ... -0.26072732 -0.26072732\n",
      "  -0.26072732]\n",
      " [-0.22885025 -0.22885025 -0.22885025 ... -0.23545645 -0.23545645\n",
      "  -0.23545645]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23843071 -0.23843071 -0.23843071 ... -0.25229278 -0.25229278\n",
      "  -0.25229278]\n",
      " [-0.2262303  -0.2262303  -0.2262303  ... -0.26249847 -0.26249847\n",
      "  -0.26249847]\n",
      " [-0.23390047 -0.23390047 -0.23390047 ... -0.20975965 -0.20975965\n",
      "  -0.20975965]\n",
      " ...\n",
      " [-0.23197415 -0.23197415 -0.23197415 ... -0.23846665 -0.23846665\n",
      "  -0.23846665]\n",
      " [-0.22237967 -0.22237967 -0.22237967 ... -0.21864593 -0.21864593\n",
      "  -0.21864593]\n",
      " [-0.21957962 -0.21957962 -0.21957962 ... -0.23479554 -0.23479554\n",
      "  -0.23479554]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22292386 -0.22292386 -0.22292386 ... -0.23218259 -0.23218259\n",
      "  -0.23218259]\n",
      " [-0.23367153 -0.23367153 -0.23367153 ... -0.23697984 -0.23697984\n",
      "  -0.23697984]\n",
      " [-0.2263788  -0.2263788  -0.2263788  ... -0.24559632 -0.24559632\n",
      "  -0.24559632]\n",
      " ...\n",
      " [-0.26868308 -0.26868308 -0.26868308 ... -0.27705932 -0.27705932\n",
      "  -0.27705932]\n",
      " [-0.26174286 -0.26174286 -0.26174286 ... -0.2488254  -0.2488254\n",
      "  -0.2488254 ]\n",
      " [-0.24431328 -0.24431328 -0.24431328 ... -0.24009597 -0.24009597\n",
      "  -0.24009597]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.23344721 -0.23344721 -0.23344721 ... -0.22406152 -0.22406152\n",
      "  -0.22406152]\n",
      " [-0.25556266 -0.25556266 -0.25556266 ... -0.21976821 -0.21976821\n",
      "  -0.21976821]\n",
      " [-0.2137346  -0.2137346  -0.2137346  ... -0.23424841 -0.23424841\n",
      "  -0.23424841]\n",
      " ...\n",
      " [-0.25580513 -0.25580513 -0.25580513 ... -0.2308574  -0.2308574\n",
      "  -0.2308574 ]\n",
      " [-0.25472528 -0.25472528 -0.25472528 ... -0.23797482 -0.23797482\n",
      "  -0.23797482]\n",
      " [-0.21145657 -0.21145657 -0.21145657 ... -0.23036447 -0.23036447\n",
      "  -0.23036447]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.24657445 -0.24657445 -0.24657445 ... -0.24862961 -0.24862961\n",
      "  -0.24862961]\n",
      " [-0.2400166  -0.2400166  -0.2400166  ... -0.23041868 -0.23041868\n",
      "  -0.23041868]\n",
      " [-0.22351798 -0.22351798 -0.22351798 ... -0.20939758 -0.20939758\n",
      "  -0.20939758]\n",
      " ...\n",
      " [-0.25058365 -0.25058365 -0.25058365 ... -0.2763397  -0.2763397\n",
      "  -0.2763397 ]\n",
      " [-0.20549498 -0.20549498 -0.20549498 ... -0.26890552 -0.26890552\n",
      "  -0.26890552]\n",
      " [-0.24604166 -0.24604166 -0.24604166 ... -0.22005278 -0.22005278\n",
      "  -0.22005278]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24148147 -0.24148147 -0.24148147 ... -0.24632737 -0.24632737\n",
      "  -0.24632737]\n",
      " [-0.2500786  -0.2500786  -0.2500786  ... -0.21109901 -0.21109901\n",
      "  -0.21109901]\n",
      " [-0.2519068  -0.2519068  -0.2519068  ... -0.24958883 -0.24958883\n",
      "  -0.24958883]\n",
      " ...\n",
      " [-0.2421723  -0.2421723  -0.2421723  ... -0.23210736 -0.23210736\n",
      "  -0.23210736]\n",
      " [-0.24686375 -0.24686375 -0.24686375 ... -0.24843478 -0.24843478\n",
      "  -0.24843478]\n",
      " [-0.23640144 -0.23640144 -0.23640144 ... -0.23828582 -0.23828582\n",
      "  -0.23828582]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.22606227 -0.22606227 -0.22606227 ... -0.21369152 -0.21369152\n",
      "  -0.21369152]\n",
      " [-0.23570296 -0.23570296 -0.23570296 ... -0.23456661 -0.23456661\n",
      "  -0.23456661]\n",
      " [-0.24094824 -0.24094824 -0.24094824 ... -0.24952963 -0.24952963\n",
      "  -0.24952963]\n",
      " ...\n",
      " [-0.2490762  -0.2490762  -0.2490762  ... -0.22032574 -0.22032574\n",
      "  -0.22032574]\n",
      " [-0.22768757 -0.22768757 -0.22768757 ... -0.20755768 -0.20755768\n",
      "  -0.20755768]\n",
      " [-0.25136042 -0.25136042 -0.25136042 ... -0.23744842 -0.23744842\n",
      "  -0.23744842]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  27 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20555727 -0.20555727 -0.20555727 ... -0.19028197 -0.19028197\n",
      "  -0.19028197]\n",
      " [-0.22184473 -0.22184473 -0.22184473 ... -0.2285952  -0.2285952\n",
      "  -0.2285952 ]\n",
      " [-0.2227237  -0.2227237  -0.2227237  ... -0.22486453 -0.22486453\n",
      "  -0.22486453]\n",
      " ...\n",
      " [-0.23515889 -0.23515889 -0.23515889 ... -0.22742909 -0.22742909\n",
      "  -0.22742909]\n",
      " [-0.22698227 -0.22698227 -0.22698227 ... -0.21967119 -0.21967119\n",
      "  -0.21967119]\n",
      " [-0.22793242 -0.22793242 -0.22793242 ... -0.15935367 -0.15935367\n",
      "  -0.15935367]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24092594 -0.24092594 -0.24092594 ... -0.2077112  -0.2077112\n",
      "  -0.2077112 ]\n",
      " [-0.21809472 -0.21809472 -0.21809472 ... -0.23377669 -0.23377669\n",
      "  -0.23377669]\n",
      " [-0.23273665 -0.23273665 -0.23273665 ... -0.21475923 -0.21475923\n",
      "  -0.21475923]\n",
      " ...\n",
      " [-0.20353404 -0.20353404 -0.20353404 ... -0.20427269 -0.20427269\n",
      "  -0.20427269]\n",
      " [-0.19970876 -0.19970876 -0.19970876 ... -0.22167236 -0.22167236\n",
      "  -0.22167236]\n",
      " [-0.21749958 -0.21749958 -0.21749958 ... -0.22801185 -0.22801185\n",
      "  -0.22801185]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20077059 -0.20077059 -0.20077059 ... -0.21634412 -0.21634412\n",
      "  -0.21634412]\n",
      " [-0.20713566 -0.20713566 -0.20713566 ... -0.20043951 -0.20043951\n",
      "  -0.20043951]\n",
      " [-0.232279   -0.232279   -0.232279   ... -0.23442899 -0.23442899\n",
      "  -0.23442899]\n",
      " ...\n",
      " [-0.22125003 -0.22125003 -0.22125003 ... -0.21060488 -0.21060488\n",
      "  -0.21060488]\n",
      " [-0.20413156 -0.20413156 -0.20413156 ... -0.21514335 -0.21514335\n",
      "  -0.21514335]\n",
      " [-0.20626506 -0.20626506 -0.20626506 ... -0.21016672 -0.21016672\n",
      "  -0.21016672]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24696545 -0.24696545 -0.24696545 ... -0.19371694 -0.19371694\n",
      "  -0.19371694]\n",
      " [-0.2230762  -0.2230762  -0.2230762  ... -0.23395102 -0.23395102\n",
      "  -0.23395102]\n",
      " [-0.23117375 -0.23117375 -0.23117375 ... -0.21078536 -0.21078536\n",
      "  -0.21078536]\n",
      " ...\n",
      " [-0.26811534 -0.26811534 -0.26811534 ... -0.2319626  -0.2319626\n",
      "  -0.2319626 ]\n",
      " [-0.24863532 -0.24863532 -0.24863532 ... -0.20632885 -0.20632885\n",
      "  -0.20632885]\n",
      " [-0.21708927 -0.21708927 -0.21708927 ... -0.20555009 -0.20555009\n",
      "  -0.20555009]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.20048021 -0.20048021 -0.20048021 ... -0.22086942 -0.22086942\n",
      "  -0.22086942]\n",
      " [-0.23741871 -0.23741871 -0.23741871 ... -0.2292653  -0.2292653\n",
      "  -0.2292653 ]\n",
      " [-0.2120923  -0.2120923  -0.2120923  ... -0.22847402 -0.22847402\n",
      "  -0.22847402]\n",
      " ...\n",
      " [-0.22345376 -0.22345376 -0.22345376 ... -0.19144726 -0.19144726\n",
      "  -0.19144726]\n",
      " [-0.20499687 -0.20499687 -0.20499687 ... -0.1737839  -0.1737839\n",
      "  -0.1737839 ]\n",
      " [-0.20129512 -0.20129512 -0.20129512 ... -0.23314399 -0.23314399\n",
      "  -0.23314399]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.23072177 -0.23072177 -0.23072177 ... -0.21724632 -0.21724632\n",
      "  -0.21724632]\n",
      " [-0.23387504 -0.23387504 -0.23387504 ... -0.22928923 -0.22928923\n",
      "  -0.22928923]\n",
      " [-0.22424892 -0.22424892 -0.22424892 ... -0.21219134 -0.21219134\n",
      "  -0.21219134]\n",
      " ...\n",
      " [-0.24294505 -0.24294505 -0.24294505 ... -0.22018766 -0.22018766\n",
      "  -0.22018766]\n",
      " [-0.23987897 -0.23987897 -0.23987897 ... -0.22340699 -0.22340699\n",
      "  -0.22340699]\n",
      " [-0.221679   -0.221679   -0.221679   ... -0.18580337 -0.18580337\n",
      "  -0.18580337]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.2221485  -0.2221485  -0.2221485  ... -0.20965321 -0.20965321\n",
      "  -0.20965321]\n",
      " [-0.19538133 -0.19538133 -0.19538133 ... -0.21726918 -0.21726918\n",
      "  -0.21726918]\n",
      " [-0.20135498 -0.20135498 -0.20135498 ... -0.2124071  -0.2124071\n",
      "  -0.2124071 ]\n",
      " ...\n",
      " [-0.22498757 -0.22498757 -0.22498757 ... -0.22813487 -0.22813487\n",
      "  -0.22813487]\n",
      " [-0.20727855 -0.20727855 -0.20727855 ... -0.22802246 -0.22802246\n",
      "  -0.22802246]\n",
      " [-0.22046639 -0.22046639 -0.22046639 ... -0.21510234 -0.21510234\n",
      "  -0.21510234]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.20359385 -0.20359385 -0.20359385 ... -0.1859279  -0.1859279\n",
      "  -0.1859279 ]\n",
      " [-0.25297976 -0.25297976 -0.25297976 ... -0.19717337 -0.19717337\n",
      "  -0.19717337]\n",
      " [-0.21242073 -0.21242073 -0.21242073 ... -0.21438214 -0.21438214\n",
      "  -0.21438214]\n",
      " ...\n",
      " [-0.21579045 -0.21579045 -0.21579045 ... -0.18616065 -0.18616065\n",
      "  -0.18616065]\n",
      " [-0.18806535 -0.18806535 -0.18806535 ... -0.20386256 -0.20386256\n",
      "  -0.20386256]\n",
      " [-0.19489384 -0.19489384 -0.19489384 ... -0.24170512 -0.24170512\n",
      "  -0.24170512]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  28 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1896804  -0.1896804  -0.1896804  ... -0.17670514 -0.17670514\n",
      "  -0.17670514]\n",
      " [-0.21344793 -0.21344793 -0.21344793 ... -0.1897239  -0.1897239\n",
      "  -0.1897239 ]\n",
      " [-0.19733383 -0.19733383 -0.19733383 ... -0.17628865 -0.17628865\n",
      "  -0.17628865]\n",
      " ...\n",
      " [-0.199254   -0.199254   -0.199254   ... -0.19903034 -0.19903034\n",
      "  -0.19903034]\n",
      " [-0.16208838 -0.16208838 -0.16208838 ... -0.19566785 -0.19566785\n",
      "  -0.19566785]\n",
      " [-0.18742242 -0.18742242 -0.18742242 ... -0.18132702 -0.18132702\n",
      "  -0.18132702]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.18553065 -0.18553065 -0.18553065 ... -0.20956054 -0.20956054\n",
      "  -0.20956054]\n",
      " [-0.20415564 -0.20415564 -0.20415564 ... -0.20312211 -0.20312211\n",
      "  -0.20312211]\n",
      " [-0.2039993  -0.2039993  -0.2039993  ... -0.18802145 -0.18802145\n",
      "  -0.18802145]\n",
      " ...\n",
      " [-0.17797412 -0.17797412 -0.17797412 ... -0.19347072 -0.19347072\n",
      "  -0.19347072]\n",
      " [-0.23552871 -0.23552871 -0.23552871 ... -0.20892465 -0.20892465\n",
      "  -0.20892465]\n",
      " [-0.19685721 -0.19685721 -0.19685721 ... -0.17345963 -0.17345963\n",
      "  -0.17345963]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21049684 -0.21049684 -0.21049684 ... -0.21802557 -0.21802557\n",
      "  -0.21802557]\n",
      " [-0.18696919 -0.18696919 -0.18696919 ... -0.23812532 -0.23812532\n",
      "  -0.23812532]\n",
      " [-0.18899593 -0.18899593 -0.18899593 ... -0.21772847 -0.21772847\n",
      "  -0.21772847]\n",
      " ...\n",
      " [-0.21154653 -0.21154653 -0.21154653 ... -0.2024045  -0.2024045\n",
      "  -0.2024045 ]\n",
      " [-0.18415582 -0.18415582 -0.18415582 ... -0.20662268 -0.20662268\n",
      "  -0.20662268]\n",
      " [-0.18914427 -0.18914427 -0.18914427 ... -0.18189272 -0.18189272\n",
      "  -0.18189272]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1821625  -0.1821625  -0.1821625  ... -0.19140576 -0.19140576\n",
      "  -0.19140576]\n",
      " [-0.2142499  -0.2142499  -0.2142499  ... -0.22587794 -0.22587794\n",
      "  -0.22587794]\n",
      " [-0.20382212 -0.20382212 -0.20382212 ... -0.20786183 -0.20786183\n",
      "  -0.20786183]\n",
      " ...\n",
      " [-0.21431291 -0.21431291 -0.21431291 ... -0.18245678 -0.18245678\n",
      "  -0.18245678]\n",
      " [-0.18516111 -0.18516111 -0.18516111 ... -0.18406758 -0.18406758\n",
      "  -0.18406758]\n",
      " [-0.21736807 -0.21736807 -0.21736807 ... -0.23143475 -0.23143475\n",
      "  -0.23143475]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.18426917 -0.18426917 -0.18426917 ... -0.16971937 -0.16971937\n",
      "  -0.16971937]\n",
      " [-0.20620441 -0.20620441 -0.20620441 ... -0.18320061 -0.18320061\n",
      "  -0.18320061]\n",
      " [-0.22202173 -0.22202173 -0.22202173 ... -0.20304242 -0.20304242\n",
      "  -0.20304242]\n",
      " ...\n",
      " [-0.20853513 -0.20853513 -0.20853513 ... -0.20008469 -0.20008469\n",
      "  -0.20008469]\n",
      " [-0.21040621 -0.21040621 -0.21040621 ... -0.18296465 -0.18296465\n",
      "  -0.18296465]\n",
      " [-0.20038688 -0.20038688 -0.20038688 ... -0.22570689 -0.22570689\n",
      "  -0.22570689]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.18987964 -0.18987964 -0.18987964 ... -0.2400092  -0.2400092\n",
      "  -0.2400092 ]\n",
      " [-0.17134278 -0.17134278 -0.17134278 ... -0.20582955 -0.20582955\n",
      "  -0.20582955]\n",
      " [-0.20441696 -0.20441696 -0.20441696 ... -0.17162709 -0.17162709\n",
      "  -0.17162709]\n",
      " ...\n",
      " [-0.1772309  -0.1772309  -0.1772309  ... -0.19343048 -0.19343048\n",
      "  -0.19343048]\n",
      " [-0.1656298  -0.1656298  -0.1656298  ... -0.20626208 -0.20626208\n",
      "  -0.20626208]\n",
      " [-0.18585086 -0.18585086 -0.18585086 ... -0.21635579 -0.21635579\n",
      "  -0.21635579]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.18287814 -0.18287814 -0.18287814 ... -0.24102831 -0.24102831\n",
      "  -0.24102831]\n",
      " [-0.19602168 -0.19602168 -0.19602168 ... -0.166417   -0.166417\n",
      "  -0.166417  ]\n",
      " [-0.22233033 -0.22233033 -0.22233033 ... -0.20652473 -0.20652473\n",
      "  -0.20652473]\n",
      " ...\n",
      " [-0.20903698 -0.20903698 -0.20903698 ... -0.24230984 -0.24230984\n",
      "  -0.24230984]\n",
      " [-0.20042269 -0.20042269 -0.20042269 ... -0.18844907 -0.18844907\n",
      "  -0.18844907]\n",
      " [-0.19846001 -0.19846001 -0.19846001 ... -0.18874209 -0.18874209\n",
      "  -0.18874209]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.1781904  -0.1781904  -0.1781904  ... -0.17719874 -0.17719874\n",
      "  -0.17719874]\n",
      " [-0.20900601 -0.20900601 -0.20900601 ... -0.21763185 -0.21763185\n",
      "  -0.21763185]\n",
      " [-0.22708422 -0.22708422 -0.22708422 ... -0.19654728 -0.19654728\n",
      "  -0.19654728]\n",
      " ...\n",
      " [-0.21839854 -0.21839854 -0.21839854 ... -0.19215944 -0.19215944\n",
      "  -0.19215944]\n",
      " [-0.21678403 -0.21678403 -0.21678403 ... -0.21030715 -0.21030715\n",
      "  -0.21030715]\n",
      " [-0.17249666 -0.17249666 -0.17249666 ... -0.17422809 -0.17422809\n",
      "  -0.17422809]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  29 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.15861729 -0.15861729 -0.15861729 ... -0.17313877 -0.17313877\n",
      "  -0.17313877]\n",
      " [-0.17519337 -0.17519337 -0.17519337 ... -0.17943217 -0.17943217\n",
      "  -0.17943217]\n",
      " [-0.20003872 -0.20003872 -0.20003872 ... -0.15819094 -0.15819094\n",
      "  -0.15819094]\n",
      " ...\n",
      " [-0.1634497  -0.1634497  -0.1634497  ... -0.15492943 -0.15492943\n",
      "  -0.15492943]\n",
      " [-0.18615887 -0.18615887 -0.18615887 ... -0.1677427  -0.1677427\n",
      "  -0.1677427 ]\n",
      " [-0.17263846 -0.17263846 -0.17263846 ... -0.16029838 -0.16029838\n",
      "  -0.16029838]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16920874 -0.16920874 -0.16920874 ... -0.18651865 -0.18651865\n",
      "  -0.18651865]\n",
      " [-0.16447632 -0.16447632 -0.16447632 ... -0.19568674 -0.19568674\n",
      "  -0.19568674]\n",
      " [-0.17824896 -0.17824896 -0.17824896 ... -0.15537924 -0.15537924\n",
      "  -0.15537924]\n",
      " ...\n",
      " [-0.166383   -0.166383   -0.166383   ... -0.19417869 -0.19417869\n",
      "  -0.19417869]\n",
      " [-0.20014006 -0.20014006 -0.20014006 ... -0.17925683 -0.17925683\n",
      "  -0.17925683]\n",
      " [-0.1701993  -0.1701993  -0.1701993  ... -0.20737305 -0.20737305\n",
      "  -0.20737305]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.19648114 -0.19648114 -0.19648114 ... -0.20925424 -0.20925424\n",
      "  -0.20925424]\n",
      " [-0.19681603 -0.19681603 -0.19681603 ... -0.20826943 -0.20826943\n",
      "  -0.20826943]\n",
      " [-0.16690794 -0.16690794 -0.16690794 ... -0.19523728 -0.19523728\n",
      "  -0.19523728]\n",
      " ...\n",
      " [-0.18470156 -0.18470156 -0.18470156 ... -0.17316012 -0.17316012\n",
      "  -0.17316012]\n",
      " [-0.20794788 -0.20794788 -0.20794788 ... -0.18095264 -0.18095264\n",
      "  -0.18095264]\n",
      " [-0.17914614 -0.17914614 -0.17914614 ... -0.12454599 -0.12454599\n",
      "  -0.12454599]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18521254 -0.18521254 -0.18521254 ... -0.18729977 -0.18729977\n",
      "  -0.18729977]\n",
      " [-0.19232564 -0.19232564 -0.19232564 ... -0.2034093  -0.2034093\n",
      "  -0.2034093 ]\n",
      " [-0.18356621 -0.18356621 -0.18356621 ... -0.17434065 -0.17434065\n",
      "  -0.17434065]\n",
      " ...\n",
      " [-0.15401435 -0.15401435 -0.15401435 ... -0.1717625  -0.1717625\n",
      "  -0.1717625 ]\n",
      " [-0.18007241 -0.18007241 -0.18007241 ... -0.17866093 -0.17866093\n",
      "  -0.17866093]\n",
      " [-0.153792   -0.153792   -0.153792   ... -0.18977281 -0.18977281\n",
      "  -0.18977281]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.1949048  -0.1949048  -0.1949048  ... -0.22081697 -0.22081697\n",
      "  -0.22081697]\n",
      " [-0.19660279 -0.19660279 -0.19660279 ... -0.19084649 -0.19084649\n",
      "  -0.19084649]\n",
      " [-0.17296782 -0.17296782 -0.17296782 ... -0.17954785 -0.17954785\n",
      "  -0.17954785]\n",
      " ...\n",
      " [-0.16095805 -0.16095805 -0.16095805 ... -0.13614316 -0.13614316\n",
      "  -0.13614316]\n",
      " [-0.1815281  -0.1815281  -0.1815281  ... -0.17947344 -0.17947344\n",
      "  -0.17947344]\n",
      " [-0.16625966 -0.16625966 -0.16625966 ... -0.17523275 -0.17523275\n",
      "  -0.17523275]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.18550017 -0.18550017 -0.18550017 ... -0.19409585 -0.19409585\n",
      "  -0.19409585]\n",
      " [-0.17946322 -0.17946322 -0.17946322 ... -0.21242431 -0.21242431\n",
      "  -0.21242431]\n",
      " [-0.154367   -0.154367   -0.154367   ... -0.20542252 -0.20542252\n",
      "  -0.20542252]\n",
      " ...\n",
      " [-0.1714631  -0.1714631  -0.1714631  ... -0.22217762 -0.22217762\n",
      "  -0.22217762]\n",
      " [-0.17600219 -0.17600219 -0.17600219 ... -0.17406029 -0.17406029\n",
      "  -0.17406029]\n",
      " [-0.16744165 -0.16744165 -0.16744165 ... -0.18382074 -0.18382074\n",
      "  -0.18382074]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.17509635 -0.17509635 -0.17509635 ... -0.18379673 -0.18379673\n",
      "  -0.18379673]\n",
      " [-0.16404708 -0.16404708 -0.16404708 ... -0.16151482 -0.16151482\n",
      "  -0.16151482]\n",
      " [-0.19231409 -0.19231409 -0.19231409 ... -0.16455626 -0.16455626\n",
      "  -0.16455626]\n",
      " ...\n",
      " [-0.1876013  -0.1876013  -0.1876013  ... -0.16368198 -0.16368198\n",
      "  -0.16368198]\n",
      " [-0.20217958 -0.20217958 -0.20217958 ... -0.19221407 -0.19221407\n",
      "  -0.19221407]\n",
      " [-0.1737415  -0.1737415  -0.1737415  ... -0.17544213 -0.17544213\n",
      "  -0.17544213]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.17014168 -0.17014168 -0.17014168 ... -0.18523218 -0.18523218\n",
      "  -0.18523218]\n",
      " [-0.19506945 -0.19506945 -0.19506945 ... -0.16871284 -0.16871284\n",
      "  -0.16871284]\n",
      " [-0.18960638 -0.18960638 -0.18960638 ... -0.15778163 -0.15778163\n",
      "  -0.15778163]\n",
      " ...\n",
      " [-0.17448376 -0.17448376 -0.17448376 ... -0.17066588 -0.17066588\n",
      "  -0.17066588]\n",
      " [-0.19266292 -0.19266292 -0.19266292 ... -0.1715532  -0.1715532\n",
      "  -0.1715532 ]\n",
      " [-0.20928168 -0.20928168 -0.20928168 ... -0.13899267 -0.13899267\n",
      "  -0.13899267]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  30 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.17542297 -0.17542297 -0.17542297 ... -0.16943279 -0.16943279\n",
      "  -0.16943279]\n",
      " [-0.17623335 -0.17623335 -0.17623335 ... -0.19091883 -0.19091883\n",
      "  -0.19091883]\n",
      " [-0.17132157 -0.17132157 -0.17132157 ... -0.15730482 -0.15730482\n",
      "  -0.15730482]\n",
      " ...\n",
      " [-0.14190184 -0.14190184 -0.14190184 ... -0.15729855 -0.15729855\n",
      "  -0.15729855]\n",
      " [-0.17775045 -0.17775045 -0.17775045 ... -0.15107052 -0.15107052\n",
      "  -0.15107052]\n",
      " [-0.18589711 -0.18589711 -0.18589711 ... -0.19471425 -0.19471425\n",
      "  -0.19471425]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.17151028 -0.17151028 -0.17151028 ... -0.14649442 -0.14649442\n",
      "  -0.14649442]\n",
      " [-0.16956015 -0.16956015 -0.16956015 ... -0.17339367 -0.17339367\n",
      "  -0.17339367]\n",
      " [-0.18287492 -0.18287492 -0.18287492 ... -0.1673139  -0.1673139\n",
      "  -0.1673139 ]\n",
      " ...\n",
      " [-0.20457551 -0.20457551 -0.20457551 ... -0.14892933 -0.14892933\n",
      "  -0.14892933]\n",
      " [-0.10289558 -0.10289558 -0.10289558 ... -0.16699874 -0.16699874\n",
      "  -0.16699874]\n",
      " [-0.16282885 -0.16282885 -0.16282885 ... -0.15505546 -0.15505546\n",
      "  -0.15505546]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14126433 -0.14126433 -0.14126433 ... -0.1590149  -0.1590149\n",
      "  -0.1590149 ]\n",
      " [-0.17412183 -0.17412183 -0.17412183 ... -0.15526691 -0.15526691\n",
      "  -0.15526691]\n",
      " [-0.13324395 -0.13324395 -0.13324395 ... -0.20749924 -0.20749924\n",
      "  -0.20749924]\n",
      " ...\n",
      " [-0.15418324 -0.15418324 -0.15418324 ... -0.15539229 -0.15539229\n",
      "  -0.15539229]\n",
      " [-0.1616692  -0.1616692  -0.1616692  ... -0.14076668 -0.14076668\n",
      "  -0.14076668]\n",
      " [-0.17063475 -0.17063475 -0.17063475 ... -0.22489227 -0.22489227\n",
      "  -0.22489227]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18162976 -0.18162976 -0.18162976 ... -0.1590184  -0.1590184\n",
      "  -0.1590184 ]\n",
      " [-0.20873265 -0.20873265 -0.20873265 ... -0.14208913 -0.14208913\n",
      "  -0.14208913]\n",
      " [-0.17914972 -0.17914972 -0.17914972 ... -0.14457975 -0.14457975\n",
      "  -0.14457975]\n",
      " ...\n",
      " [-0.14282444 -0.14282444 -0.14282444 ... -0.19148822 -0.19148822\n",
      "  -0.19148822]\n",
      " [-0.14883535 -0.14883535 -0.14883535 ... -0.14229374 -0.14229374\n",
      "  -0.14229374]\n",
      " [-0.17268279 -0.17268279 -0.17268279 ... -0.18039866 -0.18039866\n",
      "  -0.18039866]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.16218433 -0.16218433 -0.16218433 ... -0.16566633 -0.16566633\n",
      "  -0.16566633]\n",
      " [-0.16534735 -0.16534735 -0.16534735 ... -0.20112301 -0.20112301\n",
      "  -0.20112301]\n",
      " [-0.14705752 -0.14705752 -0.14705752 ... -0.18160993 -0.18160993\n",
      "  -0.18160993]\n",
      " ...\n",
      " [-0.16155292 -0.16155292 -0.16155292 ... -0.14737381 -0.14737381\n",
      "  -0.14737381]\n",
      " [-0.13861643 -0.13861643 -0.13861643 ... -0.1718974  -0.1718974\n",
      "  -0.1718974 ]\n",
      " [-0.11204074 -0.11204074 -0.11204074 ... -0.14861986 -0.14861986\n",
      "  -0.14861986]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.16185465 -0.16185465 -0.16185465 ... -0.14119306 -0.14119306\n",
      "  -0.14119306]\n",
      " [-0.17425019 -0.17425019 -0.17425019 ... -0.15195277 -0.15195277\n",
      "  -0.15195277]\n",
      " [-0.15963069 -0.15963069 -0.15963069 ... -0.1782377  -0.1782377\n",
      "  -0.1782377 ]\n",
      " ...\n",
      " [-0.18061112 -0.18061112 -0.18061112 ... -0.21716055 -0.21716055\n",
      "  -0.21716055]\n",
      " [-0.16907482 -0.16907482 -0.16907482 ... -0.15872356 -0.15872356\n",
      "  -0.15872356]\n",
      " [-0.16744371 -0.16744371 -0.16744371 ... -0.13775055 -0.13775055\n",
      "  -0.13775055]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.15728083 -0.15728083 -0.15728083 ... -0.16660038 -0.16660038\n",
      "  -0.16660038]\n",
      " [-0.1562649  -0.1562649  -0.1562649  ... -0.20262995 -0.20262995\n",
      "  -0.20262995]\n",
      " [-0.17241845 -0.17241845 -0.17241845 ... -0.18311206 -0.18311206\n",
      "  -0.18311206]\n",
      " ...\n",
      " [-0.12520045 -0.12520045 -0.12520045 ... -0.18142489 -0.18142489\n",
      "  -0.18142489]\n",
      " [-0.1484258  -0.1484258  -0.1484258  ... -0.12840584 -0.12840584\n",
      "  -0.12840584]\n",
      " [-0.15634933 -0.15634933 -0.15634933 ... -0.18825847 -0.18825847\n",
      "  -0.18825847]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.17659241 -0.17659241 -0.17659241 ... -0.14998096 -0.14998096\n",
      "  -0.14998096]\n",
      " [-0.13475847 -0.13475847 -0.13475847 ... -0.15548956 -0.15548956\n",
      "  -0.15548956]\n",
      " [-0.1566389  -0.1566389  -0.1566389  ... -0.17140523 -0.17140523\n",
      "  -0.17140523]\n",
      " ...\n",
      " [-0.15871541 -0.15871541 -0.15871541 ... -0.2297951  -0.2297951\n",
      "  -0.2297951 ]\n",
      " [-0.17501165 -0.17501165 -0.17501165 ... -0.18760276 -0.18760276\n",
      "  -0.18760276]\n",
      " [-0.18662731 -0.18662731 -0.18662731 ... -0.14018953 -0.14018953\n",
      "  -0.14018953]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  31 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13489158 -0.13489158 -0.13489158 ... -0.1470474  -0.1470474\n",
      "  -0.1470474 ]\n",
      " [-0.18425171 -0.18425171 -0.18425171 ... -0.1682307  -0.1682307\n",
      "  -0.1682307 ]\n",
      " [-0.15208045 -0.15208045 -0.15208045 ... -0.1476514  -0.1476514\n",
      "  -0.1476514 ]\n",
      " ...\n",
      " [-0.13068879 -0.13068879 -0.13068879 ... -0.15420982 -0.15420982\n",
      "  -0.15420982]\n",
      " [-0.16129337 -0.16129337 -0.16129337 ... -0.15625471 -0.15625471\n",
      "  -0.15625471]\n",
      " [-0.1267418  -0.1267418  -0.1267418  ... -0.12653184 -0.12653184\n",
      "  -0.12653184]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13761622 -0.13761622 -0.13761622 ... -0.12157506 -0.12157506\n",
      "  -0.12157506]\n",
      " [-0.12852338 -0.12852338 -0.12852338 ... -0.17084444 -0.17084444\n",
      "  -0.17084444]\n",
      " [-0.12963146 -0.12963146 -0.12963146 ... -0.14974281 -0.14974281\n",
      "  -0.14974281]\n",
      " ...\n",
      " [-0.15926045 -0.15926045 -0.15926045 ... -0.11979238 -0.11979238\n",
      "  -0.11979238]\n",
      " [-0.10691406 -0.10691406 -0.10691406 ... -0.18623205 -0.18623205\n",
      "  -0.18623205]\n",
      " [-0.15024993 -0.15024993 -0.15024993 ... -0.15181285 -0.15181285\n",
      "  -0.15181285]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13811825 -0.13811825 -0.13811825 ... -0.1447157  -0.1447157\n",
      "  -0.1447157 ]\n",
      " [-0.11966538 -0.11966538 -0.11966538 ... -0.12843566 -0.12843566\n",
      "  -0.12843566]\n",
      " [-0.13691333 -0.13691333 -0.13691333 ... -0.1456933  -0.1456933\n",
      "  -0.1456933 ]\n",
      " ...\n",
      " [-0.1288225  -0.1288225  -0.1288225  ... -0.12565139 -0.12565139\n",
      "  -0.12565139]\n",
      " [-0.13105184 -0.13105184 -0.13105184 ... -0.15641272 -0.15641272\n",
      "  -0.15641272]\n",
      " [-0.19277799 -0.19277799 -0.19277799 ... -0.11504371 -0.11504371\n",
      "  -0.11504371]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15917213 -0.15917213 -0.15917213 ... -0.14110762 -0.14110762\n",
      "  -0.14110762]\n",
      " [-0.16680335 -0.16680335 -0.16680335 ... -0.12739472 -0.12739472\n",
      "  -0.12739472]\n",
      " [-0.1771251  -0.1771251  -0.1771251  ... -0.11644318 -0.11644318\n",
      "  -0.11644318]\n",
      " ...\n",
      " [-0.14096004 -0.14096004 -0.14096004 ... -0.10935228 -0.10935228\n",
      "  -0.10935228]\n",
      " [-0.13489158 -0.13489158 -0.13489158 ... -0.14874831 -0.14874831\n",
      "  -0.14874831]\n",
      " [-0.12640552 -0.12640552 -0.12640552 ... -0.1850198  -0.1850198\n",
      "  -0.1850198 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.11179662 -0.11179662 -0.11179662 ... -0.15761371 -0.15761371\n",
      "  -0.15761371]\n",
      " [-0.1335859  -0.1335859  -0.1335859  ... -0.1716453  -0.1716453\n",
      "  -0.1716453 ]\n",
      " [-0.1487327  -0.1487327  -0.1487327  ... -0.14399235 -0.14399235\n",
      "  -0.14399235]\n",
      " ...\n",
      " [-0.16033849 -0.16033849 -0.16033849 ... -0.16895905 -0.16895905\n",
      "  -0.16895905]\n",
      " [-0.14958209 -0.14958209 -0.14958209 ... -0.15139282 -0.15139282\n",
      "  -0.15139282]\n",
      " [-0.12983133 -0.12983133 -0.12983133 ... -0.1302254  -0.1302254\n",
      "  -0.1302254 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.11937004 -0.11937004 -0.11937004 ... -0.14722279 -0.14722279\n",
      "  -0.14722279]\n",
      " [-0.1412606  -0.1412606  -0.1412606  ... -0.16128838 -0.16128838\n",
      "  -0.16128838]\n",
      " [-0.14896113 -0.14896113 -0.14896113 ... -0.12599643 -0.12599643\n",
      "  -0.12599643]\n",
      " ...\n",
      " [-0.1692437  -0.1692437  -0.1692437  ... -0.16297118 -0.16297118\n",
      "  -0.16297118]\n",
      " [-0.1432284  -0.1432284  -0.1432284  ... -0.15136307 -0.15136307\n",
      "  -0.15136307]\n",
      " [-0.12114631 -0.12114631 -0.12114631 ... -0.15767877 -0.15767877\n",
      "  -0.15767877]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.16764301 -0.16764301 -0.16764301 ... -0.15509321 -0.15509321\n",
      "  -0.15509321]\n",
      " [-0.16267565 -0.16267565 -0.16267565 ... -0.13894808 -0.13894808\n",
      "  -0.13894808]\n",
      " [-0.16143672 -0.16143672 -0.16143672 ... -0.14714217 -0.14714217\n",
      "  -0.14714217]\n",
      " ...\n",
      " [-0.16115998 -0.16115998 -0.16115998 ... -0.13379529 -0.13379529\n",
      "  -0.13379529]\n",
      " [-0.14566618 -0.14566618 -0.14566618 ... -0.12912124 -0.12912124\n",
      "  -0.12912124]\n",
      " [-0.12513015 -0.12513015 -0.12513015 ... -0.17462872 -0.17462872\n",
      "  -0.17462872]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.14735326 -0.14735326 -0.14735326 ... -0.1855751  -0.1855751\n",
      "  -0.1855751 ]\n",
      " [-0.17026687 -0.17026687 -0.17026687 ... -0.16788375 -0.16788375\n",
      "  -0.16788375]\n",
      " [-0.13761368 -0.13761368 -0.13761368 ... -0.12060769 -0.12060769\n",
      "  -0.12060769]\n",
      " ...\n",
      " [-0.11966538 -0.11966538 -0.11966538 ... -0.1322769  -0.1322769\n",
      "  -0.1322769 ]\n",
      " [-0.15965651 -0.15965651 -0.15965651 ... -0.15740073 -0.15740073\n",
      "  -0.15740073]\n",
      " [-0.18820266 -0.18820266 -0.18820266 ... -0.1395935  -0.1395935\n",
      "  -0.1395935 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  32 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1515274  -0.1515274  -0.1515274  ... -0.14542866 -0.14542866\n",
      "  -0.14542866]\n",
      " [-0.11194907 -0.11194907 -0.11194907 ... -0.15331486 -0.15331486\n",
      "  -0.15331486]\n",
      " [-0.13846326 -0.13846326 -0.13846326 ... -0.15362307 -0.15362307\n",
      "  -0.15362307]\n",
      " ...\n",
      " [-0.16697419 -0.16697419 -0.16697419 ... -0.13752745 -0.13752745\n",
      "  -0.13752745]\n",
      " [-0.14540687 -0.14540687 -0.14540687 ... -0.15984985 -0.15984985\n",
      "  -0.15984985]\n",
      " [-0.14668098 -0.14668098 -0.14668098 ... -0.12983885 -0.12983885\n",
      "  -0.12983885]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.12333038 -0.12333038 -0.12333038 ... -0.17307444 -0.17307444\n",
      "  -0.17307444]\n",
      " [-0.1086438  -0.1086438  -0.1086438  ... -0.13135241 -0.13135241\n",
      "  -0.13135241]\n",
      " [-0.16049421 -0.16049421 -0.16049421 ... -0.15183878 -0.15183878\n",
      "  -0.15183878]\n",
      " ...\n",
      " [-0.13899213 -0.13899213 -0.13899213 ... -0.1536586  -0.1536586\n",
      "  -0.1536586 ]\n",
      " [-0.13711415 -0.13711415 -0.13711415 ... -0.16826221 -0.16826221\n",
      "  -0.16826221]\n",
      " [-0.16243336 -0.16243336 -0.16243336 ... -0.13107888 -0.13107888\n",
      "  -0.13107888]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1550258  -0.1550258  -0.1550258  ... -0.16581699 -0.16581699\n",
      "  -0.16581699]\n",
      " [-0.15098757 -0.15098757 -0.15098757 ... -0.10967614 -0.10967614\n",
      "  -0.10967614]\n",
      " [-0.14496048 -0.14496048 -0.14496048 ... -0.16049269 -0.16049269\n",
      "  -0.16049269]\n",
      " ...\n",
      " [-0.11493032 -0.11493032 -0.11493032 ... -0.15212789 -0.15212789\n",
      "  -0.15212789]\n",
      " [-0.17154567 -0.17154567 -0.17154567 ... -0.12198605 -0.12198605\n",
      "  -0.12198605]\n",
      " [-0.1476683  -0.1476683  -0.1476683  ... -0.12832324 -0.12832324\n",
      "  -0.12832324]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.11193042 -0.11193042 -0.11193042 ... -0.15325849 -0.15325849\n",
      "  -0.15325849]\n",
      " [-0.17547789 -0.17547789 -0.17547789 ... -0.1680027  -0.1680027\n",
      "  -0.1680027 ]\n",
      " [-0.13525027 -0.13525027 -0.13525027 ... -0.127072   -0.127072\n",
      "  -0.127072  ]\n",
      " ...\n",
      " [-0.14815925 -0.14815925 -0.14815925 ... -0.11953519 -0.11953519\n",
      "  -0.11953519]\n",
      " [-0.15693444 -0.15693444 -0.15693444 ... -0.13131614 -0.13131614\n",
      "  -0.13131614]\n",
      " [-0.10870902 -0.10870902 -0.10870902 ... -0.15359944 -0.15359944\n",
      "  -0.15359944]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.1888635  -0.1888635  -0.1888635  ... -0.14606287 -0.14606287\n",
      "  -0.14606287]\n",
      " [-0.11020458 -0.11020458 -0.11020458 ... -0.13538918 -0.13538918\n",
      "  -0.13538918]\n",
      " [-0.14402202 -0.14402202 -0.14402202 ... -0.11754121 -0.11754121\n",
      "  -0.11754121]\n",
      " ...\n",
      " [-0.13109514 -0.13109514 -0.13109514 ... -0.12709323 -0.12709323\n",
      "  -0.12709323]\n",
      " [-0.12737472 -0.12737472 -0.12737472 ... -0.15667208 -0.15667208\n",
      "  -0.15667208]\n",
      " [-0.14243409 -0.14243409 -0.14243409 ... -0.14954412 -0.14954412\n",
      "  -0.14954412]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.13785523 -0.13785523 -0.13785523 ... -0.1745678  -0.1745678\n",
      "  -0.1745678 ]\n",
      " [-0.11933158 -0.11933158 -0.11933158 ... -0.17303073 -0.17303073\n",
      "  -0.17303073]\n",
      " [-0.17047003 -0.17047003 -0.17047003 ... -0.12618037 -0.12618037\n",
      "  -0.12618037]\n",
      " ...\n",
      " [-0.13522261 -0.13522261 -0.13522261 ... -0.14756043 -0.14756043\n",
      "  -0.14756043]\n",
      " [-0.16009346 -0.16009346 -0.16009346 ... -0.15899622 -0.15899622\n",
      "  -0.15899622]\n",
      " [-0.15573025 -0.15573025 -0.15573025 ... -0.18088314 -0.18088314\n",
      "  -0.18088314]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.14187971 -0.14187971 -0.14187971 ... -0.1538135  -0.1538135\n",
      "  -0.1538135 ]\n",
      " [-0.15260422 -0.15260422 -0.15260422 ... -0.14586106 -0.14586106\n",
      "  -0.14586106]\n",
      " [-0.15672788 -0.15672788 -0.15672788 ... -0.127334   -0.127334\n",
      "  -0.127334  ]\n",
      " ...\n",
      " [-0.16062091 -0.16062091 -0.16062091 ... -0.12772694 -0.12772694\n",
      "  -0.12772694]\n",
      " [-0.14832261 -0.14832261 -0.14832261 ... -0.1506617  -0.1506617\n",
      "  -0.1506617 ]\n",
      " [-0.11798915 -0.11798915 -0.11798915 ... -0.13451394 -0.13451394\n",
      "  -0.13451394]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.14936282 -0.14936282 -0.14936282 ... -0.12176456 -0.12176456\n",
      "  -0.12176456]\n",
      " [-0.13331312 -0.13331312 -0.13331312 ... -0.14245278 -0.14245278\n",
      "  -0.14245278]\n",
      " [-0.1339444  -0.1339444  -0.1339444  ... -0.15267591 -0.15267591\n",
      "  -0.15267591]\n",
      " ...\n",
      " [-0.1414524  -0.1414524  -0.1414524  ... -0.12047976 -0.12047976\n",
      "  -0.12047976]\n",
      " [-0.14036843 -0.14036843 -0.14036843 ... -0.13243167 -0.13243167\n",
      "  -0.13243167]\n",
      " [-0.1340824  -0.1340824  -0.1340824  ... -0.13994977 -0.13994977\n",
      "  -0.13994977]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  33 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13814428 -0.13814428 -0.13814428 ... -0.11206695 -0.11206695\n",
      "  -0.11206695]\n",
      " [-0.12238292 -0.12238292 -0.12238292 ... -0.12285443 -0.12285443\n",
      "  -0.12285443]\n",
      " [-0.14009143 -0.14009143 -0.14009143 ... -0.10607163 -0.10607163\n",
      "  -0.10607163]\n",
      " ...\n",
      " [-0.15173894 -0.15173894 -0.15173894 ... -0.1127007  -0.1127007\n",
      "  -0.1127007 ]\n",
      " [-0.14317173 -0.14317173 -0.14317173 ... -0.09148323 -0.09148323\n",
      "  -0.09148323]\n",
      " [-0.1625773  -0.1625773  -0.1625773  ... -0.21250638 -0.21250638\n",
      "  -0.21250638]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.11896247 -0.11896247 -0.11896247 ... -0.16595033 -0.16595033\n",
      "  -0.16595033]\n",
      " [-0.13442677 -0.13442677 -0.13442677 ... -0.09462631 -0.09462631\n",
      "  -0.09462631]\n",
      " [-0.13375436 -0.13375436 -0.13375436 ... -0.14526528 -0.14526528\n",
      "  -0.14526528]\n",
      " ...\n",
      " [-0.12723884 -0.12723884 -0.12723884 ... -0.14185122 -0.14185122\n",
      "  -0.14185122]\n",
      " [-0.14178775 -0.14178775 -0.14178775 ... -0.15837708 -0.15837708\n",
      "  -0.15837708]\n",
      " [-0.14096405 -0.14096405 -0.14096405 ... -0.1496766  -0.1496766\n",
      "  -0.1496766 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.06337278 -0.06337278 -0.06337278 ... -0.14304686 -0.14304686\n",
      "  -0.14304686]\n",
      " [-0.1409499  -0.1409499  -0.1409499  ... -0.16519235 -0.16519235\n",
      "  -0.16519235]\n",
      " [-0.14400293 -0.14400293 -0.14400293 ... -0.16475327 -0.16475327\n",
      "  -0.16475327]\n",
      " ...\n",
      " [-0.14617586 -0.14617586 -0.14617586 ... -0.1444235  -0.1444235\n",
      "  -0.1444235 ]\n",
      " [-0.14478008 -0.14478008 -0.14478008 ... -0.15126216 -0.15126216\n",
      "  -0.15126216]\n",
      " [-0.1395415  -0.1395415  -0.1395415  ... -0.15168369 -0.15168369\n",
      "  -0.15168369]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.12989382 -0.12989382 -0.12989382 ... -0.16324344 -0.16324344\n",
      "  -0.16324344]\n",
      " [-0.14044562 -0.14044562 -0.14044562 ... -0.14296041 -0.14296041\n",
      "  -0.14296041]\n",
      " [-0.13969766 -0.13969766 -0.13969766 ... -0.14598665 -0.14598665\n",
      "  -0.14598665]\n",
      " ...\n",
      " [-0.15001404 -0.15001404 -0.15001404 ... -0.15686789 -0.15686789\n",
      "  -0.15686789]\n",
      " [-0.1402829  -0.1402829  -0.1402829  ... -0.17518616 -0.17518616\n",
      "  -0.17518616]\n",
      " [-0.11586479 -0.11586479 -0.11586479 ... -0.13839921 -0.13839921\n",
      "  -0.13839921]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.13229142 -0.13229142 -0.13229142 ... -0.13576241 -0.13576241\n",
      "  -0.13576241]\n",
      " [-0.11465284 -0.11465284 -0.11465284 ... -0.16491196 -0.16491196\n",
      "  -0.16491196]\n",
      " [-0.16472611 -0.16472611 -0.16472611 ... -0.15859945 -0.15859945\n",
      "  -0.15859945]\n",
      " ...\n",
      " [-0.12630823 -0.12630823 -0.12630823 ... -0.13873404 -0.13873404\n",
      "  -0.13873404]\n",
      " [-0.13569425 -0.13569425 -0.13569425 ... -0.16187498 -0.16187498\n",
      "  -0.16187498]\n",
      " [-0.17122671 -0.17122671 -0.17122671 ... -0.15605202 -0.15605202\n",
      "  -0.15605202]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.12394983 -0.12394983 -0.12394983 ... -0.13942836 -0.13942836\n",
      "  -0.13942836]\n",
      " [-0.12153452 -0.12153452 -0.12153452 ... -0.13394034 -0.13394034\n",
      "  -0.13394034]\n",
      " [-0.1391062  -0.1391062  -0.1391062  ... -0.16152789 -0.16152789\n",
      "  -0.16152789]\n",
      " ...\n",
      " [-0.14309198 -0.14309198 -0.14309198 ... -0.16546449 -0.16546449\n",
      "  -0.16546449]\n",
      " [-0.13023518 -0.13023518 -0.13023518 ... -0.14862514 -0.14862514\n",
      "  -0.14862514]\n",
      " [-0.15364522 -0.15364522 -0.15364522 ... -0.13958575 -0.13958575\n",
      "  -0.13958575]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.1267716  -0.1267716  -0.1267716  ... -0.1411758  -0.1411758\n",
      "  -0.1411758 ]\n",
      " [-0.10789033 -0.10789033 -0.10789033 ... -0.12459821 -0.12459821\n",
      "  -0.12459821]\n",
      " [-0.14098664 -0.14098664 -0.14098664 ... -0.13754043 -0.13754043\n",
      "  -0.13754043]\n",
      " ...\n",
      " [-0.14452861 -0.14452861 -0.14452861 ... -0.1434744  -0.1434744\n",
      "  -0.1434744 ]\n",
      " [-0.1368405  -0.1368405  -0.1368405  ... -0.1157821  -0.1157821\n",
      "  -0.1157821 ]\n",
      " [-0.1081315  -0.1081315  -0.1081315  ... -0.13525921 -0.13525921\n",
      "  -0.13525921]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.12017629 -0.12017629 -0.12017629 ... -0.08987309 -0.08987309\n",
      "  -0.08987309]\n",
      " [-0.10954995 -0.10954995 -0.10954995 ... -0.14248326 -0.14248326\n",
      "  -0.14248326]\n",
      " [-0.12603642 -0.12603642 -0.12603642 ... -0.14379825 -0.14379825\n",
      "  -0.14379825]\n",
      " ...\n",
      " [-0.13616619 -0.13616619 -0.13616619 ... -0.13831508 -0.13831508\n",
      "  -0.13831508]\n",
      " [-0.09200717 -0.09200717 -0.09200717 ... -0.15567456 -0.15567456\n",
      "  -0.15567456]\n",
      " [-0.10312276 -0.10312276 -0.10312276 ... -0.1368482  -0.1368482\n",
      "  -0.1368482 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  34 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13053373 -0.13053373 -0.13053373 ... -0.16829923 -0.16829923\n",
      "  -0.16829923]\n",
      " [-0.12267791 -0.12267791 -0.12267791 ... -0.13874787 -0.13874787\n",
      "  -0.13874787]\n",
      " [-0.15151384 -0.15151384 -0.15151384 ... -0.09777969 -0.09777969\n",
      "  -0.09777969]\n",
      " ...\n",
      " [-0.1267448  -0.1267448  -0.1267448  ... -0.10234334 -0.10234334\n",
      "  -0.10234334]\n",
      " [-0.16809106 -0.16809106 -0.16809106 ... -0.16737422 -0.16737422\n",
      "  -0.16737422]\n",
      " [-0.14517534 -0.14517534 -0.14517534 ... -0.1307665  -0.1307665\n",
      "  -0.1307665 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.08599437 -0.08599437 -0.08599437 ... -0.1546773  -0.1546773\n",
      "  -0.1546773 ]\n",
      " [-0.13887423 -0.13887423 -0.13887423 ... -0.20112933 -0.20112933\n",
      "  -0.20112933]\n",
      " [-0.16667657 -0.16667657 -0.16667657 ... -0.1563475  -0.1563475\n",
      "  -0.1563475 ]\n",
      " ...\n",
      " [-0.09819798 -0.09819798 -0.09819798 ... -0.14136678 -0.14136678\n",
      "  -0.14136678]\n",
      " [-0.14631833 -0.14631833 -0.14631833 ... -0.13795254 -0.13795254\n",
      "  -0.13795254]\n",
      " [-0.14553379 -0.14553379 -0.14553379 ... -0.13740544 -0.13740544\n",
      "  -0.13740544]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13277327 -0.13277327 -0.13277327 ... -0.14347647 -0.14347647\n",
      "  -0.14347647]\n",
      " [-0.1098361  -0.1098361  -0.1098361  ... -0.13011415 -0.13011415\n",
      "  -0.13011415]\n",
      " [-0.1413099  -0.1413099  -0.1413099  ... -0.13371065 -0.13371065\n",
      "  -0.13371065]\n",
      " ...\n",
      " [-0.17250095 -0.17250095 -0.17250095 ... -0.10779679 -0.10779679\n",
      "  -0.10779679]\n",
      " [-0.15192509 -0.15192509 -0.15192509 ... -0.1509972  -0.1509972\n",
      "  -0.1509972 ]\n",
      " [-0.14259006 -0.14259006 -0.14259006 ... -0.12478345 -0.12478345\n",
      "  -0.12478345]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15244345 -0.15244345 -0.15244345 ... -0.12741564 -0.12741564\n",
      "  -0.12741564]\n",
      " [-0.09825335 -0.09825335 -0.09825335 ... -0.12680104 -0.12680104\n",
      "  -0.12680104]\n",
      " [-0.10748941 -0.10748941 -0.10748941 ... -0.13504715 -0.13504715\n",
      "  -0.13504715]\n",
      " ...\n",
      " [-0.12758598 -0.12758598 -0.12758598 ... -0.1261534  -0.1261534\n",
      "  -0.1261534 ]\n",
      " [-0.13613527 -0.13613527 -0.13613527 ... -0.12055287 -0.12055287\n",
      "  -0.12055287]\n",
      " [-0.14666164 -0.14666164 -0.14666164 ... -0.13646373 -0.13646373\n",
      "  -0.13646373]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.14256364 -0.14256364 -0.14256364 ... -0.11833623 -0.11833623\n",
      "  -0.11833623]\n",
      " [-0.12219591 -0.12219591 -0.12219591 ... -0.12660453 -0.12660453\n",
      "  -0.12660453]\n",
      " [-0.14288151 -0.14288151 -0.14288151 ... -0.12048709 -0.12048709\n",
      "  -0.12048709]\n",
      " ...\n",
      " [-0.10680895 -0.10680895 -0.10680895 ... -0.11255091 -0.11255091\n",
      "  -0.11255091]\n",
      " [-0.16864076 -0.16864076 -0.16864076 ... -0.13206258 -0.13206258\n",
      "  -0.13206258]\n",
      " [-0.12609056 -0.12609056 -0.12609056 ... -0.10731331 -0.10731331\n",
      "  -0.10731331]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.16763878 -0.16763878 -0.16763878 ... -0.14101511 -0.14101511\n",
      "  -0.14101511]\n",
      " [-0.15523832 -0.15523832 -0.15523832 ... -0.12612267 -0.12612267\n",
      "  -0.12612267]\n",
      " [-0.12741658 -0.12741658 -0.12741658 ... -0.13134855 -0.13134855\n",
      "  -0.13134855]\n",
      " ...\n",
      " [-0.13305739 -0.13305739 -0.13305739 ... -0.1554279  -0.1554279\n",
      "  -0.1554279 ]\n",
      " [-0.14390425 -0.14390425 -0.14390425 ... -0.12973751 -0.12973751\n",
      "  -0.12973751]\n",
      " [-0.16422886 -0.16422886 -0.16422886 ... -0.11508437 -0.11508437\n",
      "  -0.11508437]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.14029017 -0.14029017 -0.14029017 ... -0.13192087 -0.13192087\n",
      "  -0.13192087]\n",
      " [-0.12332013 -0.12332013 -0.12332013 ... -0.13591826 -0.13591826\n",
      "  -0.13591826]\n",
      " [-0.17912154 -0.17912154 -0.17912154 ... -0.12683472 -0.12683472\n",
      "  -0.12683472]\n",
      " ...\n",
      " [-0.12454994 -0.12454994 -0.12454994 ... -0.14788738 -0.14788738\n",
      "  -0.14788738]\n",
      " [-0.15985128 -0.15985128 -0.15985128 ... -0.0979474  -0.0979474\n",
      "  -0.0979474 ]\n",
      " [-0.10574101 -0.10574101 -0.10574101 ... -0.14429913 -0.14429913\n",
      "  -0.14429913]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.12639701 -0.12639701 -0.12639701 ... -0.10639613 -0.10639613\n",
      "  -0.10639613]\n",
      " [-0.15349871 -0.15349871 -0.15349871 ... -0.13486272 -0.13486272\n",
      "  -0.13486272]\n",
      " [-0.13750046 -0.13750046 -0.13750046 ... -0.16504744 -0.16504744\n",
      "  -0.16504744]\n",
      " ...\n",
      " [-0.15401524 -0.15401524 -0.15401524 ... -0.16860583 -0.16860583\n",
      "  -0.16860583]\n",
      " [-0.16689545 -0.16689545 -0.16689545 ... -0.15233251 -0.15233251\n",
      "  -0.15233251]\n",
      " [-0.15299204 -0.15299204 -0.15299204 ... -0.13712302 -0.13712302\n",
      "  -0.13712302]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  35 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.12746397 -0.12746397 -0.12746397 ... -0.15914893 -0.15914893\n",
      "  -0.15914893]\n",
      " [-0.12941106 -0.12941106 -0.12941106 ... -0.13059062 -0.13059062\n",
      "  -0.13059062]\n",
      " [-0.12511975 -0.12511975 -0.12511975 ... -0.15752535 -0.15752535\n",
      "  -0.15752535]\n",
      " ...\n",
      " [-0.14748184 -0.14748184 -0.14748184 ... -0.0896081  -0.0896081\n",
      "  -0.0896081 ]\n",
      " [-0.13866112 -0.13866112 -0.13866112 ... -0.16066279 -0.16066279\n",
      "  -0.16066279]\n",
      " [-0.14856052 -0.14856052 -0.14856052 ... -0.1766639  -0.1766639\n",
      "  -0.1766639 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1547895  -0.1547895  -0.1547895  ... -0.11945394 -0.11945394\n",
      "  -0.11945394]\n",
      " [-0.10561858 -0.10561858 -0.10561858 ... -0.16103764 -0.16103764\n",
      "  -0.16103764]\n",
      " [-0.16387293 -0.16387293 -0.16387293 ... -0.12596156 -0.12596156\n",
      "  -0.12596156]\n",
      " ...\n",
      " [-0.12066957 -0.12066957 -0.12066957 ... -0.18187846 -0.18187846\n",
      "  -0.18187846]\n",
      " [-0.09529614 -0.09529614 -0.09529614 ... -0.15031402 -0.15031402\n",
      "  -0.15031402]\n",
      " [-0.14380641 -0.14380641 -0.14380641 ... -0.13294044 -0.13294044\n",
      "  -0.13294044]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.19406821 -0.19406821 -0.19406821 ... -0.12738508 -0.12738508\n",
      "  -0.12738508]\n",
      " [-0.12409227 -0.12409227 -0.12409227 ... -0.150586   -0.150586\n",
      "  -0.150586  ]\n",
      " [-0.1459024  -0.1459024  -0.1459024  ... -0.13085522 -0.13085522\n",
      "  -0.13085522]\n",
      " ...\n",
      " [-0.13739064 -0.13739064 -0.13739064 ... -0.14193124 -0.14193124\n",
      "  -0.14193124]\n",
      " [-0.12379141 -0.12379141 -0.12379141 ... -0.1617471  -0.1617471\n",
      "  -0.1617471 ]\n",
      " [-0.1385434  -0.1385434  -0.1385434  ... -0.12822473 -0.12822473\n",
      "  -0.12822473]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15754625 -0.15754625 -0.15754625 ... -0.16281703 -0.16281703\n",
      "  -0.16281703]\n",
      " [-0.14587891 -0.14587891 -0.14587891 ... -0.16327521 -0.16327521\n",
      "  -0.16327521]\n",
      " [-0.10834785 -0.10834785 -0.10834785 ... -0.11977708 -0.11977708\n",
      "  -0.11977708]\n",
      " ...\n",
      " [-0.14463627 -0.14463627 -0.14463627 ... -0.11802549 -0.11802549\n",
      "  -0.11802549]\n",
      " [-0.1352977  -0.1352977  -0.1352977  ... -0.1366348  -0.1366348\n",
      "  -0.1366348 ]\n",
      " [-0.12425879 -0.12425879 -0.12425879 ... -0.12542486 -0.12542486\n",
      "  -0.12542486]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.1285238  -0.1285238  -0.1285238  ... -0.15225753 -0.15225753\n",
      "  -0.15225753]\n",
      " [-0.09831882 -0.09831882 -0.09831882 ... -0.11518826 -0.11518826\n",
      "  -0.11518826]\n",
      " [-0.12492855 -0.12492855 -0.12492855 ... -0.13491929 -0.13491929\n",
      "  -0.13491929]\n",
      " ...\n",
      " [-0.13799667 -0.13799667 -0.13799667 ... -0.10949276 -0.10949276\n",
      "  -0.10949276]\n",
      " [-0.14651413 -0.14651413 -0.14651413 ... -0.12682697 -0.12682697\n",
      "  -0.12682697]\n",
      " [-0.17469364 -0.17469364 -0.17469364 ... -0.12657353 -0.12657353\n",
      "  -0.12657353]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.16275097 -0.16275097 -0.16275097 ... -0.15854886 -0.15854886\n",
      "  -0.15854886]\n",
      " [-0.12140115 -0.12140115 -0.12140115 ... -0.14365298 -0.14365298\n",
      "  -0.14365298]\n",
      " [-0.11870296 -0.11870296 -0.11870296 ... -0.14636199 -0.14636199\n",
      "  -0.14636199]\n",
      " ...\n",
      " [-0.13204381 -0.13204381 -0.13204381 ... -0.16578358 -0.16578358\n",
      "  -0.16578358]\n",
      " [-0.13684157 -0.13684157 -0.13684157 ... -0.17007834 -0.17007834\n",
      "  -0.17007834]\n",
      " [-0.10964413 -0.10964413 -0.10964413 ... -0.15364194 -0.15364194\n",
      "  -0.15364194]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.14616339 -0.14616339 -0.14616339 ... -0.15369007 -0.15369007\n",
      "  -0.15369007]\n",
      " [-0.16459078 -0.16459078 -0.16459078 ... -0.14880867 -0.14880867\n",
      "  -0.14880867]\n",
      " [-0.13035065 -0.13035065 -0.13035065 ... -0.14543012 -0.14543012\n",
      "  -0.14543012]\n",
      " ...\n",
      " [-0.14413269 -0.14413269 -0.14413269 ... -0.10795432 -0.10795432\n",
      "  -0.10795432]\n",
      " [-0.15332839 -0.15332839 -0.15332839 ... -0.1502875  -0.1502875\n",
      "  -0.1502875 ]\n",
      " [-0.1115597  -0.1115597  -0.1115597  ... -0.08394951 -0.08394951\n",
      "  -0.08394951]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.17952527 -0.17952527 -0.17952527 ... -0.14824276 -0.14824276\n",
      "  -0.14824276]\n",
      " [-0.12749794 -0.12749794 -0.12749794 ... -0.13343038 -0.13343038\n",
      "  -0.13343038]\n",
      " [-0.14670649 -0.14670649 -0.14670649 ... -0.10232143 -0.10232143\n",
      "  -0.10232143]\n",
      " ...\n",
      " [-0.13562468 -0.13562468 -0.13562468 ... -0.14323393 -0.14323393\n",
      "  -0.14323393]\n",
      " [-0.12997371 -0.12997371 -0.12997371 ... -0.11998588 -0.11998588\n",
      "  -0.11998588]\n",
      " [-0.15612285 -0.15612285 -0.15612285 ... -0.09071037 -0.09071037\n",
      "  -0.09071037]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  36 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.06405328 -0.06405328 -0.06405328 ... -0.13683882 -0.13683882\n",
      "  -0.13683882]\n",
      " [-0.12286855 -0.12286855 -0.12286855 ... -0.10786526 -0.10786526\n",
      "  -0.10786526]\n",
      " [-0.14316519 -0.14316519 -0.14316519 ... -0.12777558 -0.12777558\n",
      "  -0.12777558]\n",
      " ...\n",
      " [-0.1289829  -0.1289829  -0.1289829  ... -0.12792471 -0.12792471\n",
      "  -0.12792471]\n",
      " [-0.15074895 -0.15074895 -0.15074895 ... -0.1170946  -0.1170946\n",
      "  -0.1170946 ]\n",
      " [-0.1457327  -0.1457327  -0.1457327  ... -0.13504033 -0.13504033\n",
      "  -0.13504033]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.11454844 -0.11454844 -0.11454844 ... -0.14470962 -0.14470962\n",
      "  -0.14470962]\n",
      " [-0.08909258 -0.08909258 -0.08909258 ... -0.17917094 -0.17917094\n",
      "  -0.17917094]\n",
      " [-0.13585643 -0.13585643 -0.13585643 ... -0.16493043 -0.16493043\n",
      "  -0.16493043]\n",
      " ...\n",
      " [-0.16052023 -0.16052023 -0.16052023 ... -0.12015436 -0.12015436\n",
      "  -0.12015436]\n",
      " [-0.14152801 -0.14152801 -0.14152801 ... -0.13933    -0.13933\n",
      "  -0.13933   ]\n",
      " [-0.15940823 -0.15940823 -0.15940823 ... -0.13407995 -0.13407995\n",
      "  -0.13407995]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15251477 -0.15251477 -0.15251477 ... -0.163296   -0.163296\n",
      "  -0.163296  ]\n",
      " [-0.13630392 -0.13630392 -0.13630392 ... -0.16678675 -0.16678675\n",
      "  -0.16678675]\n",
      " [-0.18468875 -0.18468875 -0.18468875 ... -0.15260005 -0.15260005\n",
      "  -0.15260005]\n",
      " ...\n",
      " [-0.12208833 -0.12208833 -0.12208833 ... -0.16707636 -0.16707636\n",
      "  -0.16707636]\n",
      " [-0.13287434 -0.13287434 -0.13287434 ... -0.1396856  -0.1396856\n",
      "  -0.1396856 ]\n",
      " [-0.13846214 -0.13846214 -0.13846214 ... -0.16997755 -0.16997755\n",
      "  -0.16997755]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.10337213 -0.10337213 -0.10337213 ... -0.16157083 -0.16157083\n",
      "  -0.16157083]\n",
      " [-0.1345652  -0.1345652  -0.1345652  ... -0.1353415  -0.1353415\n",
      "  -0.1353415 ]\n",
      " [-0.1255338  -0.1255338  -0.1255338  ... -0.15795709 -0.15795709\n",
      "  -0.15795709]\n",
      " ...\n",
      " [-0.09618138 -0.09618138 -0.09618138 ... -0.16401672 -0.16401672\n",
      "  -0.16401672]\n",
      " [-0.15956613 -0.15956613 -0.15956613 ... -0.13272084 -0.13272084\n",
      "  -0.13272084]\n",
      " [-0.12823477 -0.12823477 -0.12823477 ... -0.15278448 -0.15278448\n",
      "  -0.15278448]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.13939732 -0.13939732 -0.13939732 ... -0.18085724 -0.18085724\n",
      "  -0.18085724]\n",
      " [-0.15116814 -0.15116814 -0.15116814 ... -0.12571847 -0.12571847\n",
      "  -0.12571847]\n",
      " [-0.12627415 -0.12627415 -0.12627415 ... -0.10698459 -0.10698459\n",
      "  -0.10698459]\n",
      " ...\n",
      " [-0.1471337  -0.1471337  -0.1471337  ... -0.12971692 -0.12971692\n",
      "  -0.12971692]\n",
      " [-0.12597273 -0.12597273 -0.12597273 ... -0.1498811  -0.1498811\n",
      "  -0.1498811 ]\n",
      " [-0.11105711 -0.11105711 -0.11105711 ... -0.16898923 -0.16898923\n",
      "  -0.16898923]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.12802993 -0.12802993 -0.12802993 ... -0.11548415 -0.11548415\n",
      "  -0.11548415]\n",
      " [-0.13250087 -0.13250087 -0.13250087 ... -0.19002178 -0.19002178\n",
      "  -0.19002178]\n",
      " [-0.11803719 -0.11803719 -0.11803719 ... -0.12337781 -0.12337781\n",
      "  -0.12337781]\n",
      " ...\n",
      " [-0.1544342  -0.1544342  -0.1544342  ... -0.16790307 -0.16790307\n",
      "  -0.16790307]\n",
      " [-0.13548198 -0.13548198 -0.13548198 ... -0.14396098 -0.14396098\n",
      "  -0.14396098]\n",
      " [-0.16658078 -0.16658078 -0.16658078 ... -0.11988953 -0.11988953\n",
      "  -0.11988953]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.16311976 -0.16311976 -0.16311976 ... -0.10843387 -0.10843387\n",
      "  -0.10843387]\n",
      " [-0.11496802 -0.11496802 -0.11496802 ... -0.15573642 -0.15573642\n",
      "  -0.15573642]\n",
      " [-0.16618398 -0.16618398 -0.16618398 ... -0.1396344  -0.1396344\n",
      "  -0.1396344 ]\n",
      " ...\n",
      " [-0.1421844  -0.1421844  -0.1421844  ... -0.15808244 -0.15808244\n",
      "  -0.15808244]\n",
      " [-0.1347156  -0.1347156  -0.1347156  ... -0.14631584 -0.14631584\n",
      "  -0.14631584]\n",
      " [-0.15106238 -0.15106238 -0.15106238 ... -0.13579026 -0.13579026\n",
      "  -0.13579026]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.15774462 -0.15774462 -0.15774462 ... -0.14778192 -0.14778192\n",
      "  -0.14778192]\n",
      " [-0.10693117 -0.10693117 -0.10693117 ... -0.16326982 -0.16326982\n",
      "  -0.16326982]\n",
      " [-0.12765506 -0.12765506 -0.12765506 ... -0.17516391 -0.17516391\n",
      "  -0.17516391]\n",
      " ...\n",
      " [-0.15150432 -0.15150432 -0.15150432 ... -0.11348011 -0.11348011\n",
      "  -0.11348011]\n",
      " [-0.14378226 -0.14378226 -0.14378226 ... -0.10156434 -0.10156434\n",
      "  -0.10156434]\n",
      " [-0.14632219 -0.14632219 -0.14632219 ... -0.13601297 -0.13601297\n",
      "  -0.13601297]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  37 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.15233448 -0.15233448 -0.15233448 ... -0.08264522 -0.08264522\n",
      "  -0.08264522]\n",
      " [-0.11638392 -0.11638392 -0.11638392 ... -0.16643313 -0.16643313\n",
      "  -0.16643313]\n",
      " [-0.13167141 -0.13167141 -0.13167141 ... -0.19494523 -0.19494523\n",
      "  -0.19494523]\n",
      " ...\n",
      " [-0.16210763 -0.16210763 -0.16210763 ... -0.16268376 -0.16268376\n",
      "  -0.16268376]\n",
      " [-0.14507955 -0.14507955 -0.14507955 ... -0.17138147 -0.17138147\n",
      "  -0.17138147]\n",
      " [-0.1381462  -0.1381462  -0.1381462  ... -0.1292548  -0.1292548\n",
      "  -0.1292548 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19300658 -0.19300658 -0.19300658 ... -0.14939162 -0.14939162\n",
      "  -0.14939162]\n",
      " [-0.1499458  -0.1499458  -0.1499458  ... -0.16180193 -0.16180193\n",
      "  -0.16180193]\n",
      " [-0.14093895 -0.14093895 -0.14093895 ... -0.14193682 -0.14193682\n",
      "  -0.14193682]\n",
      " ...\n",
      " [-0.1691334  -0.1691334  -0.1691334  ... -0.09705589 -0.09705589\n",
      "  -0.09705589]\n",
      " [-0.16883068 -0.16883068 -0.16883068 ... -0.15680529 -0.15680529\n",
      "  -0.15680529]\n",
      " [-0.17111388 -0.17111388 -0.17111388 ... -0.13027413 -0.13027413\n",
      "  -0.13027413]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13532814 -0.13532814 -0.13532814 ... -0.1710894  -0.1710894\n",
      "  -0.1710894 ]\n",
      " [-0.17042959 -0.17042959 -0.17042959 ... -0.1164771  -0.1164771\n",
      "  -0.1164771 ]\n",
      " [-0.12104306 -0.12104306 -0.12104306 ... -0.15106572 -0.15106572\n",
      "  -0.15106572]\n",
      " ...\n",
      " [-0.17281353 -0.17281353 -0.17281353 ... -0.10766385 -0.10766385\n",
      "  -0.10766385]\n",
      " [-0.1687443  -0.1687443  -0.1687443  ... -0.15339509 -0.15339509\n",
      "  -0.15339509]\n",
      " [-0.13241422 -0.13241422 -0.13241422 ... -0.11882637 -0.11882637\n",
      "  -0.11882637]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.16352108 -0.16352108 -0.16352108 ... -0.12388479 -0.12388479\n",
      "  -0.12388479]\n",
      " [-0.1563818  -0.1563818  -0.1563818  ... -0.14938602 -0.14938602\n",
      "  -0.14938602]\n",
      " [-0.15716864 -0.15716864 -0.15716864 ... -0.12738384 -0.12738384\n",
      "  -0.12738384]\n",
      " ...\n",
      " [-0.14082067 -0.14082067 -0.14082067 ... -0.15933977 -0.15933977\n",
      "  -0.15933977]\n",
      " [-0.15361175 -0.15361175 -0.15361175 ... -0.11873862 -0.11873862\n",
      "  -0.11873862]\n",
      " [-0.19084173 -0.19084173 -0.19084173 ... -0.10820234 -0.10820234\n",
      "  -0.10820234]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.13987628 -0.13987628 -0.13987628 ... -0.14812621 -0.14812621\n",
      "  -0.14812621]\n",
      " [-0.15975156 -0.15975156 -0.15975156 ... -0.14424168 -0.14424168\n",
      "  -0.14424168]\n",
      " [-0.20563361 -0.20563361 -0.20563361 ... -0.13263538 -0.13263538\n",
      "  -0.13263538]\n",
      " ...\n",
      " [-0.14303717 -0.14303717 -0.14303717 ... -0.16377565 -0.16377565\n",
      "  -0.16377565]\n",
      " [-0.1883445  -0.1883445  -0.1883445  ... -0.14898823 -0.14898823\n",
      "  -0.14898823]\n",
      " [-0.15641408 -0.15641408 -0.15641408 ... -0.15817037 -0.15817037\n",
      "  -0.15817037]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.12724674 -0.12724674 -0.12724674 ... -0.15609923 -0.15609923\n",
      "  -0.15609923]\n",
      " [-0.13986945 -0.13986945 -0.13986945 ... -0.10728208 -0.10728208\n",
      "  -0.10728208]\n",
      " [-0.1229105  -0.1229105  -0.1229105  ... -0.13053161 -0.13053161\n",
      "  -0.13053161]\n",
      " ...\n",
      " [-0.15035863 -0.15035863 -0.15035863 ... -0.15607375 -0.15607375\n",
      "  -0.15607375]\n",
      " [-0.13486998 -0.13486998 -0.13486998 ... -0.19309458 -0.19309458\n",
      "  -0.19309458]\n",
      " [-0.12688652 -0.12688652 -0.12688652 ... -0.13782206 -0.13782206\n",
      "  -0.13782206]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.1584694  -0.1584694  -0.1584694  ... -0.15984981 -0.15984981\n",
      "  -0.15984981]\n",
      " [-0.15302315 -0.15302315 -0.15302315 ... -0.15467651 -0.15467651\n",
      "  -0.15467651]\n",
      " [-0.14093906 -0.14093906 -0.14093906 ... -0.13553822 -0.13553822\n",
      "  -0.13553822]\n",
      " ...\n",
      " [-0.13454469 -0.13454469 -0.13454469 ... -0.17262122 -0.17262122\n",
      "  -0.17262122]\n",
      " [-0.11405024 -0.11405024 -0.11405024 ... -0.16952187 -0.16952187\n",
      "  -0.16952187]\n",
      " [-0.15736383 -0.15736383 -0.15736383 ... -0.16543907 -0.16543907\n",
      "  -0.16543907]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.13920337 -0.13920337 -0.13920337 ... -0.14580916 -0.14580916\n",
      "  -0.14580916]\n",
      " [-0.14393342 -0.14393342 -0.14393342 ... -0.15659562 -0.15659562\n",
      "  -0.15659562]\n",
      " [-0.1299799  -0.1299799  -0.1299799  ... -0.1437372  -0.1437372\n",
      "  -0.1437372 ]\n",
      " ...\n",
      " [-0.12885076 -0.12885076 -0.12885076 ... -0.15672238 -0.15672238\n",
      "  -0.15672238]\n",
      " [-0.14663452 -0.14663452 -0.14663452 ... -0.16730997 -0.16730997\n",
      "  -0.16730997]\n",
      " [-0.13203812 -0.13203812 -0.13203812 ... -0.13065974 -0.13065974\n",
      "  -0.13065974]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  38 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.10463198 -0.10463198 -0.10463198 ... -0.16859077 -0.16859077\n",
      "  -0.16859077]\n",
      " [-0.18326606 -0.18326606 -0.18326606 ... -0.153182   -0.153182\n",
      "  -0.153182  ]\n",
      " [-0.13773112 -0.13773112 -0.13773112 ... -0.15307528 -0.15307528\n",
      "  -0.15307528]\n",
      " ...\n",
      " [-0.20615347 -0.20615347 -0.20615347 ... -0.172165   -0.172165\n",
      "  -0.172165  ]\n",
      " [-0.17421284 -0.17421284 -0.17421284 ... -0.17899726 -0.17899726\n",
      "  -0.17899726]\n",
      " [-0.12812185 -0.12812185 -0.12812185 ... -0.13657543 -0.13657543\n",
      "  -0.13657543]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.17891292 -0.17891292 -0.17891292 ... -0.14485744 -0.14485744\n",
      "  -0.14485744]\n",
      " [-0.15008801 -0.15008801 -0.15008801 ... -0.17702065 -0.17702065\n",
      "  -0.17702065]\n",
      " [-0.1377445  -0.1377445  -0.1377445  ... -0.15741232 -0.15741232\n",
      "  -0.15741232]\n",
      " ...\n",
      " [-0.13904417 -0.13904417 -0.13904417 ... -0.15257436 -0.15257436\n",
      "  -0.15257436]\n",
      " [-0.1299228  -0.1299228  -0.1299228  ... -0.17805421 -0.17805421\n",
      "  -0.17805421]\n",
      " [-0.19555318 -0.19555318 -0.19555318 ... -0.12887917 -0.12887917\n",
      "  -0.12887917]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14248335 -0.14248335 -0.14248335 ... -0.13424724 -0.13424724\n",
      "  -0.13424724]\n",
      " [-0.12671204 -0.12671204 -0.12671204 ... -0.14492121 -0.14492121\n",
      "  -0.14492121]\n",
      " [-0.15097448 -0.15097448 -0.15097448 ... -0.1518726  -0.1518726\n",
      "  -0.1518726 ]\n",
      " ...\n",
      " [-0.17891979 -0.17891979 -0.17891979 ... -0.11681197 -0.11681197\n",
      "  -0.11681197]\n",
      " [-0.15328774 -0.15328774 -0.15328774 ... -0.16641226 -0.16641226\n",
      "  -0.16641226]\n",
      " [-0.17893864 -0.17893864 -0.17893864 ... -0.14333446 -0.14333446\n",
      "  -0.14333446]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.17285301 -0.17285301 -0.17285301 ... -0.19420572 -0.19420572\n",
      "  -0.19420572]\n",
      " [-0.15484957 -0.15484957 -0.15484957 ... -0.18750843 -0.18750843\n",
      "  -0.18750843]\n",
      " [-0.17535205 -0.17535205 -0.17535205 ... -0.15826768 -0.15826768\n",
      "  -0.15826768]\n",
      " ...\n",
      " [-0.15482679 -0.15482679 -0.15482679 ... -0.14199276 -0.14199276\n",
      "  -0.14199276]\n",
      " [-0.12795739 -0.12795739 -0.12795739 ... -0.14001924 -0.14001924\n",
      "  -0.14001924]\n",
      " [-0.17094406 -0.17094406 -0.17094406 ... -0.16355875 -0.16355875\n",
      "  -0.16355875]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.16005248 -0.16005248 -0.16005248 ... -0.16594768 -0.16594768\n",
      "  -0.16594768]\n",
      " [-0.13192661 -0.13192661 -0.13192661 ... -0.16478673 -0.16478673\n",
      "  -0.16478673]\n",
      " [-0.15723918 -0.15723918 -0.15723918 ... -0.15613972 -0.15613972\n",
      "  -0.15613972]\n",
      " ...\n",
      " [-0.17108986 -0.17108986 -0.17108986 ... -0.1834269  -0.1834269\n",
      "  -0.1834269 ]\n",
      " [-0.1341575  -0.1341575  -0.1341575  ... -0.17899726 -0.17899726\n",
      "  -0.17899726]\n",
      " [-0.16083838 -0.16083838 -0.16083838 ... -0.13148043 -0.13148043\n",
      "  -0.13148043]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.10831921 -0.10831921 -0.10831921 ... -0.1653158  -0.1653158\n",
      "  -0.1653158 ]\n",
      " [-0.12630486 -0.12630486 -0.12630486 ... -0.14958893 -0.14958893\n",
      "  -0.14958893]\n",
      " [-0.17176765 -0.17176765 -0.17176765 ... -0.12209906 -0.12209906\n",
      "  -0.12209906]\n",
      " ...\n",
      " [-0.14006591 -0.14006591 -0.14006591 ... -0.12679502 -0.12679502\n",
      "  -0.12679502]\n",
      " [-0.17546675 -0.17546675 -0.17546675 ... -0.13089222 -0.13089222\n",
      "  -0.13089222]\n",
      " [-0.17079641 -0.17079641 -0.17079641 ... -0.14316419 -0.14316419\n",
      "  -0.14316419]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.14981103 -0.14981103 -0.14981103 ... -0.14381364 -0.14381364\n",
      "  -0.14381364]\n",
      " [-0.16706938 -0.16706938 -0.16706938 ... -0.12166958 -0.12166958\n",
      "  -0.12166958]\n",
      " [-0.16567715 -0.16567715 -0.16567715 ... -0.16095437 -0.16095437\n",
      "  -0.16095437]\n",
      " ...\n",
      " [-0.15367143 -0.15367143 -0.15367143 ... -0.12807679 -0.12807679\n",
      "  -0.12807679]\n",
      " [-0.14824632 -0.14824632 -0.14824632 ... -0.16630012 -0.16630012\n",
      "  -0.16630012]\n",
      " [-0.13745207 -0.13745207 -0.13745207 ... -0.14089191 -0.14089191\n",
      "  -0.14089191]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.14540802 -0.14540802 -0.14540802 ... -0.13121374 -0.13121374\n",
      "  -0.13121374]\n",
      " [-0.1555919  -0.1555919  -0.1555919  ... -0.2066551  -0.2066551\n",
      "  -0.2066551 ]\n",
      " [-0.13089147 -0.13089147 -0.13089147 ... -0.15543433 -0.15543433\n",
      "  -0.15543433]\n",
      " ...\n",
      " [-0.18454754 -0.18454754 -0.18454754 ... -0.11253296 -0.11253296\n",
      "  -0.11253296]\n",
      " [-0.14871773 -0.14871773 -0.14871773 ... -0.1770373  -0.1770373\n",
      "  -0.1770373 ]\n",
      " [-0.14880404 -0.14880404 -0.14880404 ... -0.15077357 -0.15077357\n",
      "  -0.15077357]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  39 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.17007548 -0.17007548 -0.17007548 ... -0.18682477 -0.18682477\n",
      "  -0.18682477]\n",
      " [-0.16701373 -0.16701373 -0.16701373 ... -0.17348698 -0.17348698\n",
      "  -0.17348698]\n",
      " [-0.14860713 -0.14860713 -0.14860713 ... -0.21928123 -0.21928123\n",
      "  -0.21928123]\n",
      " ...\n",
      " [-0.17925362 -0.17925362 -0.17925362 ... -0.16345371 -0.16345371\n",
      "  -0.16345371]\n",
      " [-0.13801399 -0.13801399 -0.13801399 ... -0.14903954 -0.14903954\n",
      "  -0.14903954]\n",
      " [-0.16427913 -0.16427913 -0.16427913 ... -0.16820279 -0.16820279\n",
      "  -0.16820279]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.15286082 -0.15286082 -0.15286082 ... -0.15780401 -0.15780401\n",
      "  -0.15780401]\n",
      " [-0.15111133 -0.15111133 -0.15111133 ... -0.17629406 -0.17629406\n",
      "  -0.17629406]\n",
      " [-0.16747221 -0.16747221 -0.16747221 ... -0.1607847  -0.1607847\n",
      "  -0.1607847 ]\n",
      " ...\n",
      " [-0.1264892  -0.1264892  -0.1264892  ... -0.17244434 -0.17244434\n",
      "  -0.17244434]\n",
      " [-0.1466293  -0.1466293  -0.1466293  ... -0.17804089 -0.17804089\n",
      "  -0.17804089]\n",
      " [-0.15762171 -0.15762171 -0.15762171 ... -0.1526763  -0.1526763\n",
      "  -0.1526763 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.17007113 -0.17007113 -0.17007113 ... -0.15237895 -0.15237895\n",
      "  -0.15237895]\n",
      " [-0.1109045  -0.1109045  -0.1109045  ... -0.14465062 -0.14465062\n",
      "  -0.14465062]\n",
      " [-0.15320772 -0.15320772 -0.15320772 ... -0.16603108 -0.16603108\n",
      "  -0.16603108]\n",
      " ...\n",
      " [-0.16719323 -0.16719323 -0.16719323 ... -0.16116574 -0.16116574\n",
      "  -0.16116574]\n",
      " [-0.15707538 -0.15707538 -0.15707538 ... -0.12019236 -0.12019236\n",
      "  -0.12019236]\n",
      " [-0.17381045 -0.17381045 -0.17381045 ... -0.1669881  -0.1669881\n",
      "  -0.1669881 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1998231  -0.1998231  -0.1998231  ... -0.25998938 -0.25998938\n",
      "  -0.25998938]\n",
      " [-0.13399771 -0.13399771 -0.13399771 ... -0.18127011 -0.18127011\n",
      "  -0.18127011]\n",
      " [-0.14704834 -0.14704834 -0.14704834 ... -0.17429018 -0.17429018\n",
      "  -0.17429018]\n",
      " ...\n",
      " [-0.14810428 -0.14810428 -0.14810428 ... -0.14864817 -0.14864817\n",
      "  -0.14864817]\n",
      " [-0.1636964  -0.1636964  -0.1636964  ... -0.1657934  -0.1657934\n",
      "  -0.1657934 ]\n",
      " [-0.1764448  -0.1764448  -0.1764448  ... -0.14903954 -0.14903954\n",
      "  -0.14903954]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.16522554 -0.16522554 -0.16522554 ... -0.18020105 -0.18020105\n",
      "  -0.18020105]\n",
      " [-0.16975322 -0.16975322 -0.16975322 ... -0.17632884 -0.17632884\n",
      "  -0.17632884]\n",
      " [-0.14167406 -0.14167406 -0.14167406 ... -0.1751732  -0.1751732\n",
      "  -0.1751732 ]\n",
      " ...\n",
      " [-0.15507917 -0.15507917 -0.15507917 ... -0.19721873 -0.19721873\n",
      "  -0.19721873]\n",
      " [-0.10751712 -0.10751712 -0.10751712 ... -0.1751597  -0.1751597\n",
      "  -0.1751597 ]\n",
      " [-0.15054013 -0.15054013 -0.15054013 ... -0.1644308  -0.1644308\n",
      "  -0.1644308 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.1675237  -0.1675237  -0.1675237  ... -0.15903084 -0.15903084\n",
      "  -0.15903084]\n",
      " [-0.16669291 -0.16669291 -0.16669291 ... -0.13397638 -0.13397638\n",
      "  -0.13397638]\n",
      " [-0.17187843 -0.17187843 -0.17187843 ... -0.17784873 -0.17784873\n",
      "  -0.17784873]\n",
      " ...\n",
      " [-0.1658993  -0.1658993  -0.1658993  ... -0.17174177 -0.17174177\n",
      "  -0.17174177]\n",
      " [-0.1652464  -0.1652464  -0.1652464  ... -0.17559288 -0.17559288\n",
      "  -0.17559288]\n",
      " [-0.16761759 -0.16761759 -0.16761759 ... -0.16152391 -0.16152391\n",
      "  -0.16152391]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.17644948 -0.17644948 -0.17644948 ... -0.15807505 -0.15807505\n",
      "  -0.15807505]\n",
      " [-0.18894476 -0.18894476 -0.18894476 ... -0.1596797  -0.1596797\n",
      "  -0.1596797 ]\n",
      " [-0.12428052 -0.12428052 -0.12428052 ... -0.1699439  -0.1699439\n",
      "  -0.1699439 ]\n",
      " ...\n",
      " [-0.18231717 -0.18231717 -0.18231717 ... -0.16167752 -0.16167752\n",
      "  -0.16167752]\n",
      " [-0.13678706 -0.13678706 -0.13678706 ... -0.18871877 -0.18871877\n",
      "  -0.18871877]\n",
      " [-0.16706474 -0.16706474 -0.16706474 ... -0.16206674 -0.16206674\n",
      "  -0.16206674]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.18148535 -0.18148535 -0.18148535 ... -0.13555023 -0.13555023\n",
      "  -0.13555023]\n",
      " [-0.14964728 -0.14964728 -0.14964728 ... -0.15053493 -0.15053493\n",
      "  -0.15053493]\n",
      " [-0.15048784 -0.15048784 -0.15048784 ... -0.17612384 -0.17612384\n",
      "  -0.17612384]\n",
      " ...\n",
      " [-0.17054662 -0.17054662 -0.17054662 ... -0.17246874 -0.17246874\n",
      "  -0.17246874]\n",
      " [-0.153079   -0.153079   -0.153079   ... -0.14759043 -0.14759043\n",
      "  -0.14759043]\n",
      " [-0.18582407 -0.18582407 -0.18582407 ... -0.20524746 -0.20524746\n",
      "  -0.20524746]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  40 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16127078 -0.16127078 -0.16127078 ... -0.18304515 -0.18304515\n",
      "  -0.18304515]\n",
      " [-0.1922871  -0.1922871  -0.1922871  ... -0.14632958 -0.14632958\n",
      "  -0.14632958]\n",
      " [-0.1974465  -0.1974465  -0.1974465  ... -0.15940674 -0.15940674\n",
      "  -0.15940674]\n",
      " ...\n",
      " [-0.17839476 -0.17839476 -0.17839476 ... -0.23484176 -0.23484176\n",
      "  -0.23484176]\n",
      " [-0.1537202  -0.1537202  -0.1537202  ... -0.17789915 -0.17789915\n",
      "  -0.17789915]\n",
      " [-0.16126381 -0.16126381 -0.16126381 ... -0.19991472 -0.19991472\n",
      "  -0.19991472]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.15530169 -0.15530169 -0.15530169 ... -0.15739058 -0.15739058\n",
      "  -0.15739058]\n",
      " [-0.18126467 -0.18126467 -0.18126467 ... -0.1831001  -0.1831001\n",
      "  -0.1831001 ]\n",
      " [-0.16429624 -0.16429624 -0.16429624 ... -0.1760344  -0.1760344\n",
      "  -0.1760344 ]\n",
      " ...\n",
      " [-0.15476167 -0.15476167 -0.15476167 ... -0.17273518 -0.17273518\n",
      "  -0.17273518]\n",
      " [-0.17860538 -0.17860538 -0.17860538 ... -0.17742646 -0.17742646\n",
      "  -0.17742646]\n",
      " [-0.12928112 -0.12928112 -0.12928112 ... -0.15042916 -0.15042916\n",
      "  -0.15042916]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18464065 -0.18464065 -0.18464065 ... -0.16184938 -0.16184938\n",
      "  -0.16184938]\n",
      " [-0.17019174 -0.17019174 -0.17019174 ... -0.18184675 -0.18184675\n",
      "  -0.18184675]\n",
      " [-0.17860314 -0.17860314 -0.17860314 ... -0.145374   -0.145374\n",
      "  -0.145374  ]\n",
      " ...\n",
      " [-0.15260246 -0.15260246 -0.15260246 ... -0.22174281 -0.22174281\n",
      "  -0.22174281]\n",
      " [-0.19514927 -0.19514927 -0.19514927 ... -0.21115278 -0.21115278\n",
      "  -0.21115278]\n",
      " [-0.16472524 -0.16472524 -0.16472524 ... -0.21853389 -0.21853389\n",
      "  -0.21853389]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.21325722 -0.21325722 -0.21325722 ... -0.18923554 -0.18923554\n",
      "  -0.18923554]\n",
      " [-0.17944178 -0.17944178 -0.17944178 ... -0.15830109 -0.15830109\n",
      "  -0.15830109]\n",
      " [-0.13547604 -0.13547604 -0.13547604 ... -0.23312235 -0.23312235\n",
      "  -0.23312235]\n",
      " ...\n",
      " [-0.19546102 -0.19546102 -0.19546102 ... -0.18177417 -0.18177417\n",
      "  -0.18177417]\n",
      " [-0.16715443 -0.16715443 -0.16715443 ... -0.16026425 -0.16026425\n",
      "  -0.16026425]\n",
      " [-0.18812212 -0.18812212 -0.18812212 ... -0.1805385  -0.1805385\n",
      "  -0.1805385 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.16863899 -0.16863899 -0.16863899 ... -0.18765865 -0.18765865\n",
      "  -0.18765865]\n",
      " [-0.17516771 -0.17516771 -0.17516771 ... -0.18401337 -0.18401337\n",
      "  -0.18401337]\n",
      " [-0.14691563 -0.14691563 -0.14691563 ... -0.16350979 -0.16350979\n",
      "  -0.16350979]\n",
      " ...\n",
      " [-0.19142133 -0.19142133 -0.19142133 ... -0.15539925 -0.15539925\n",
      "  -0.15539925]\n",
      " [-0.2040697  -0.2040697  -0.2040697  ... -0.18052901 -0.18052901\n",
      "  -0.18052901]\n",
      " [-0.1915369  -0.1915369  -0.1915369  ... -0.19157359 -0.19157359\n",
      "  -0.19157359]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.1142696  -0.1142696  -0.1142696  ... -0.17650628 -0.17650628\n",
      "  -0.17650628]\n",
      " [-0.1775181  -0.1775181  -0.1775181  ... -0.17555502 -0.17555502\n",
      "  -0.17555502]\n",
      " [-0.14577055 -0.14577055 -0.14577055 ... -0.18477614 -0.18477614\n",
      "  -0.18477614]\n",
      " ...\n",
      " [-0.23743233 -0.23743233 -0.23743233 ... -0.17764239 -0.17764239\n",
      "  -0.17764239]\n",
      " [-0.18468802 -0.18468802 -0.18468802 ... -0.16691318 -0.16691318\n",
      "  -0.16691318]\n",
      " [-0.11360671 -0.11360671 -0.11360671 ... -0.16689259 -0.16689259\n",
      "  -0.16689259]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.17800984 -0.17800984 -0.17800984 ... -0.20362663 -0.20362663\n",
      "  -0.20362663]\n",
      " [-0.21841946 -0.21841946 -0.21841946 ... -0.20056944 -0.20056944\n",
      "  -0.20056944]\n",
      " [-0.22210681 -0.22210681 -0.22210681 ... -0.14915784 -0.14915784\n",
      "  -0.14915784]\n",
      " ...\n",
      " [-0.11237001 -0.11237001 -0.11237001 ... -0.15505144 -0.15505144\n",
      "  -0.15505144]\n",
      " [-0.15593848 -0.15593848 -0.15593848 ... -0.16171089 -0.16171089\n",
      "  -0.16171089]\n",
      " [-0.14779623 -0.14779623 -0.14779623 ... -0.22548158 -0.22548158\n",
      "  -0.22548158]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.16630858 -0.16630858 -0.16630858 ... -0.14766991 -0.14766991\n",
      "  -0.14766991]\n",
      " [-0.1707118  -0.1707118  -0.1707118  ... -0.18009001 -0.18009001\n",
      "  -0.18009001]\n",
      " [-0.14911026 -0.14911026 -0.14911026 ... -0.17711383 -0.17711383\n",
      "  -0.17711383]\n",
      " ...\n",
      " [-0.18762901 -0.18762901 -0.18762901 ... -0.18792191 -0.18792191\n",
      "  -0.18792191]\n",
      " [-0.15618192 -0.15618192 -0.15618192 ... -0.23731546 -0.23731546\n",
      "  -0.23731546]\n",
      " [-0.20557258 -0.20557258 -0.20557258 ... -0.1811523  -0.1811523\n",
      "  -0.1811523 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  41 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16766712 -0.16766712 -0.16766712 ... -0.17466736 -0.17466736\n",
      "  -0.17466736]\n",
      " [-0.19552556 -0.19552556 -0.19552556 ... -0.17731884 -0.17731884\n",
      "  -0.17731884]\n",
      " [-0.2599837  -0.2599837  -0.2599837  ... -0.16868901 -0.16868901\n",
      "  -0.16868901]\n",
      " ...\n",
      " [-0.17461368 -0.17461368 -0.17461368 ... -0.1496146  -0.1496146\n",
      "  -0.1496146 ]\n",
      " [-0.19117308 -0.19117308 -0.19117308 ... -0.18332762 -0.18332762\n",
      "  -0.18332762]\n",
      " [-0.15041094 -0.15041094 -0.15041094 ... -0.14839593 -0.14839593\n",
      "  -0.14839593]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16286656 -0.16286656 -0.16286656 ... -0.17917776 -0.17917776\n",
      "  -0.17917776]\n",
      " [-0.19911447 -0.19911447 -0.19911447 ... -0.21339057 -0.21339057\n",
      "  -0.21339057]\n",
      " [-0.2089813  -0.2089813  -0.2089813  ... -0.18628648 -0.18628648\n",
      "  -0.18628648]\n",
      " ...\n",
      " [-0.17001657 -0.17001657 -0.17001657 ... -0.18410322 -0.18410322\n",
      "  -0.18410322]\n",
      " [-0.24696937 -0.24696937 -0.24696937 ... -0.17235938 -0.17235938\n",
      "  -0.17235938]\n",
      " [-0.21156038 -0.21156038 -0.21156038 ... -0.16239634 -0.16239634\n",
      "  -0.16239634]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2141966  -0.2141966  -0.2141966  ... -0.1476749  -0.1476749\n",
      "  -0.1476749 ]\n",
      " [-0.19930923 -0.19930923 -0.19930923 ... -0.18409707 -0.18409707\n",
      "  -0.18409707]\n",
      " [-0.1597994  -0.1597994  -0.1597994  ... -0.16260208 -0.16260208\n",
      "  -0.16260208]\n",
      " ...\n",
      " [-0.19544938 -0.19544938 -0.19544938 ... -0.21811245 -0.21811245\n",
      "  -0.21811245]\n",
      " [-0.18606004 -0.18606004 -0.18606004 ... -0.13541424 -0.13541424\n",
      "  -0.13541424]\n",
      " [-0.2229715  -0.2229715  -0.2229715  ... -0.19204943 -0.19204943\n",
      "  -0.19204943]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1934402  -0.1934402  -0.1934402  ... -0.21918753 -0.21918753\n",
      "  -0.21918753]\n",
      " [-0.19003338 -0.19003338 -0.19003338 ... -0.15839398 -0.15839398\n",
      "  -0.15839398]\n",
      " [-0.18665387 -0.18665387 -0.18665387 ... -0.16264537 -0.16264537\n",
      "  -0.16264537]\n",
      " ...\n",
      " [-0.17733678 -0.17733678 -0.17733678 ... -0.1655842  -0.1655842\n",
      "  -0.1655842 ]\n",
      " [-0.1863625  -0.1863625  -0.1863625  ... -0.19047478 -0.19047478\n",
      "  -0.19047478]\n",
      " [-0.164673   -0.164673   -0.164673   ... -0.16582821 -0.16582821\n",
      "  -0.16582821]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.23303416 -0.23303416 -0.23303416 ... -0.16221511 -0.16221511\n",
      "  -0.16221511]\n",
      " [-0.20682931 -0.20682931 -0.20682931 ... -0.13242495 -0.13242495\n",
      "  -0.13242495]\n",
      " [-0.2167583  -0.2167583  -0.2167583  ... -0.2183875  -0.2183875\n",
      "  -0.2183875 ]\n",
      " ...\n",
      " [-0.19000967 -0.19000967 -0.19000967 ... -0.20453706 -0.20453706\n",
      "  -0.20453706]\n",
      " [-0.19505847 -0.19505847 -0.19505847 ... -0.17534497 -0.17534497\n",
      "  -0.17534497]\n",
      " [-0.1878622  -0.1878622  -0.1878622  ... -0.16353184 -0.16353184\n",
      "  -0.16353184]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.2117579  -0.2117579  -0.2117579  ... -0.18927917 -0.18927917\n",
      "  -0.18927917]\n",
      " [-0.19356179 -0.19356179 -0.19356179 ... -0.23929878 -0.23929878\n",
      "  -0.23929878]\n",
      " [-0.16986126 -0.16986126 -0.16986126 ... -0.20515607 -0.20515607\n",
      "  -0.20515607]\n",
      " ...\n",
      " [-0.1567176  -0.1567176  -0.1567176  ... -0.17478207 -0.17478207\n",
      "  -0.17478207]\n",
      " [-0.17874625 -0.17874625 -0.17874625 ... -0.18281487 -0.18281487\n",
      "  -0.18281487]\n",
      " [-0.2001842  -0.2001842  -0.2001842  ... -0.17253059 -0.17253059\n",
      "  -0.17253059]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.19138336 -0.19138336 -0.19138336 ... -0.13985507 -0.13985507\n",
      "  -0.13985507]\n",
      " [-0.18703806 -0.18703806 -0.18703806 ... -0.2246434  -0.2246434\n",
      "  -0.2246434 ]\n",
      " [-0.16463014 -0.16463014 -0.16463014 ... -0.19375882 -0.19375882\n",
      "  -0.19375882]\n",
      " ...\n",
      " [-0.2108636  -0.2108636  -0.2108636  ... -0.18350735 -0.18350735\n",
      "  -0.18350735]\n",
      " [-0.20629814 -0.20629814 -0.20629814 ... -0.22335495 -0.22335495\n",
      "  -0.22335495]\n",
      " [-0.20708269 -0.20708269 -0.20708269 ... -0.2092676  -0.2092676\n",
      "  -0.2092676 ]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.18036982 -0.18036982 -0.18036982 ... -0.17760661 -0.17760661\n",
      "  -0.17760661]\n",
      " [-0.17043397 -0.17043397 -0.17043397 ... -0.22298603 -0.22298603\n",
      "  -0.22298603]\n",
      " [-0.17659324 -0.17659324 -0.17659324 ... -0.18143553 -0.18143553\n",
      "  -0.18143553]\n",
      " ...\n",
      " [-0.2059026  -0.2059026  -0.2059026  ... -0.20094416 -0.20094416\n",
      "  -0.20094416]\n",
      " [-0.18584496 -0.18584496 -0.18584496 ... -0.17014034 -0.17014034\n",
      "  -0.17014034]\n",
      " [-0.23589759 -0.23589759 -0.23589759 ... -0.16029465 -0.16029465\n",
      "  -0.16029465]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  42 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20266193 -0.20266193 -0.20266193 ... -0.17308879 -0.17308879\n",
      "  -0.17308879]\n",
      " [-0.16007192 -0.16007192 -0.16007192 ... -0.20874861 -0.20874861\n",
      "  -0.20874861]\n",
      " [-0.22338642 -0.22338642 -0.22338642 ... -0.19393001 -0.19393001\n",
      "  -0.19393001]\n",
      " ...\n",
      " [-0.18364424 -0.18364424 -0.18364424 ... -0.21327041 -0.21327041\n",
      "  -0.21327041]\n",
      " [-0.19914481 -0.19914481 -0.19914481 ... -0.19567479 -0.19567479\n",
      "  -0.19567479]\n",
      " [-0.26409876 -0.26409876 -0.26409876 ... -0.20579208 -0.20579208\n",
      "  -0.20579208]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.20218132 -0.20218132 -0.20218132 ... -0.21695873 -0.21695873\n",
      "  -0.21695873]\n",
      " [-0.20778793 -0.20778793 -0.20778793 ... -0.20841631 -0.20841631\n",
      "  -0.20841631]\n",
      " [-0.18760791 -0.18760791 -0.18760791 ... -0.2056439  -0.2056439\n",
      "  -0.2056439 ]\n",
      " ...\n",
      " [-0.23459375 -0.23459375 -0.23459375 ... -0.19046655 -0.19046655\n",
      "  -0.19046655]\n",
      " [-0.19236761 -0.19236761 -0.19236761 ... -0.189038   -0.189038\n",
      "  -0.189038  ]\n",
      " [-0.19892475 -0.19892475 -0.19892475 ... -0.19300409 -0.19300409\n",
      "  -0.19300409]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21567541 -0.21567541 -0.21567541 ... -0.25048488 -0.25048488\n",
      "  -0.25048488]\n",
      " [-0.20586178 -0.20586178 -0.20586178 ... -0.19708726 -0.19708726\n",
      "  -0.19708726]\n",
      " [-0.17481545 -0.17481545 -0.17481545 ... -0.167029   -0.167029\n",
      "  -0.167029  ]\n",
      " ...\n",
      " [-0.19759344 -0.19759344 -0.19759344 ... -0.21613118 -0.21613118\n",
      "  -0.21613118]\n",
      " [-0.23515043 -0.23515043 -0.23515043 ... -0.19978231 -0.19978231\n",
      "  -0.19978231]\n",
      " [-0.23233491 -0.23233491 -0.23233491 ... -0.19000337 -0.19000337\n",
      "  -0.19000337]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2071291  -0.2071291  -0.2071291  ... -0.2266235  -0.2266235\n",
      "  -0.2266235 ]\n",
      " [-0.20310801 -0.20310801 -0.20310801 ... -0.19312927 -0.19312927\n",
      "  -0.19312927]\n",
      " [-0.17411464 -0.17411464 -0.17411464 ... -0.18349509 -0.18349509\n",
      "  -0.18349509]\n",
      " ...\n",
      " [-0.21898407 -0.21898407 -0.21898407 ... -0.1769967  -0.1769967\n",
      "  -0.1769967 ]\n",
      " [-0.18596055 -0.18596055 -0.18596055 ... -0.2105728  -0.2105728\n",
      "  -0.2105728 ]\n",
      " [-0.21163592 -0.21163592 -0.21163592 ... -0.16475017 -0.16475017\n",
      "  -0.16475017]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.18062441 -0.18062441 -0.18062441 ... -0.20408452 -0.20408452\n",
      "  -0.20408452]\n",
      " [-0.20750698 -0.20750698 -0.20750698 ... -0.22525212 -0.22525212\n",
      "  -0.22525212]\n",
      " [-0.18537411 -0.18537411 -0.18537411 ... -0.23635803 -0.23635803\n",
      "  -0.23635803]\n",
      " ...\n",
      " [-0.1508029  -0.1508029  -0.1508029  ... -0.18228285 -0.18228285\n",
      "  -0.18228285]\n",
      " [-0.20624328 -0.20624328 -0.20624328 ... -0.19108132 -0.19108132\n",
      "  -0.19108132]\n",
      " [-0.21046779 -0.21046779 -0.21046779 ... -0.1911732  -0.1911732\n",
      "  -0.1911732 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.22555344 -0.22555344 -0.22555344 ... -0.21832341 -0.21832341\n",
      "  -0.21832341]\n",
      " [-0.19245237 -0.19245237 -0.19245237 ... -0.23106746 -0.23106746\n",
      "  -0.23106746]\n",
      " [-0.21069735 -0.21069735 -0.21069735 ... -0.19240218 -0.19240218\n",
      "  -0.19240218]\n",
      " ...\n",
      " [-0.24771948 -0.24771948 -0.24771948 ... -0.21965483 -0.21965483\n",
      "  -0.21965483]\n",
      " [-0.2185676  -0.2185676  -0.2185676  ... -0.20425057 -0.20425057\n",
      "  -0.20425057]\n",
      " [-0.18991491 -0.18991491 -0.18991491 ... -0.15965034 -0.15965034\n",
      "  -0.15965034]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.19122568 -0.19122568 -0.19122568 ... -0.18018615 -0.18018615\n",
      "  -0.18018615]\n",
      " [-0.18447205 -0.18447205 -0.18447205 ... -0.20454016 -0.20454016\n",
      "  -0.20454016]\n",
      " [-0.20486893 -0.20486893 -0.20486893 ... -0.19757336 -0.19757336\n",
      "  -0.19757336]\n",
      " ...\n",
      " [-0.21489978 -0.21489978 -0.21489978 ... -0.15350986 -0.15350986\n",
      "  -0.15350986]\n",
      " [-0.19104844 -0.19104844 -0.19104844 ... -0.20297517 -0.20297517\n",
      "  -0.20297517]\n",
      " [-0.2214438  -0.2214438  -0.2214438  ... -0.18698224 -0.18698224\n",
      "  -0.18698224]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.20539916 -0.20539916 -0.20539916 ... -0.19138256 -0.19138256\n",
      "  -0.19138256]\n",
      " [-0.2057498  -0.2057498  -0.2057498  ... -0.22021744 -0.22021744\n",
      "  -0.22021744]\n",
      " [-0.14021967 -0.14021967 -0.14021967 ... -0.19784236 -0.19784236\n",
      "  -0.19784236]\n",
      " ...\n",
      " [-0.2004117  -0.2004117  -0.2004117  ... -0.16158602 -0.16158602\n",
      "  -0.16158602]\n",
      " [-0.2132085  -0.2132085  -0.2132085  ... -0.17678946 -0.17678946\n",
      "  -0.17678946]\n",
      " [-0.16867182 -0.16867182 -0.16867182 ... -0.19761834 -0.19761834\n",
      "  -0.19761834]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  43 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.26314867 -0.26314867 -0.26314867 ... -0.20632544 -0.20632544\n",
      "  -0.20632544]\n",
      " [-0.19328836 -0.19328836 -0.19328836 ... -0.20625512 -0.20625512\n",
      "  -0.20625512]\n",
      " [-0.23012602 -0.23012602 -0.23012602 ... -0.24423975 -0.24423975\n",
      "  -0.24423975]\n",
      " ...\n",
      " [-0.20555475 -0.20555475 -0.20555475 ... -0.19923252 -0.19923252\n",
      "  -0.19923252]\n",
      " [-0.1856734  -0.1856734  -0.1856734  ... -0.23535249 -0.23535249\n",
      "  -0.23535249]\n",
      " [-0.23283342 -0.23283342 -0.23283342 ... -0.26176178 -0.26176178\n",
      "  -0.26176178]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.20639557 -0.20639557 -0.20639557 ... -0.21152446 -0.21152446\n",
      "  -0.21152446]\n",
      " [-0.2145968  -0.2145968  -0.2145968  ... -0.23204948 -0.23204948\n",
      "  -0.23204948]\n",
      " [-0.21218586 -0.21218586 -0.21218586 ... -0.18016288 -0.18016288\n",
      "  -0.18016288]\n",
      " ...\n",
      " [-0.25658032 -0.25658032 -0.25658032 ... -0.2459086  -0.2459086\n",
      "  -0.2459086 ]\n",
      " [-0.22464359 -0.22464359 -0.22464359 ... -0.21996823 -0.21996823\n",
      "  -0.21996823]\n",
      " [-0.23766614 -0.23766614 -0.23766614 ... -0.25086564 -0.25086564\n",
      "  -0.25086564]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23018928 -0.23018928 -0.23018928 ... -0.20031111 -0.20031111\n",
      "  -0.20031111]\n",
      " [-0.256302   -0.256302   -0.256302   ... -0.23648481 -0.23648481\n",
      "  -0.23648481]\n",
      " [-0.22955415 -0.22955415 -0.22955415 ... -0.18835777 -0.18835777\n",
      "  -0.18835777]\n",
      " ...\n",
      " [-0.17436925 -0.17436925 -0.17436925 ... -0.2270785  -0.2270785\n",
      "  -0.2270785 ]\n",
      " [-0.19400778 -0.19400778 -0.19400778 ... -0.20072363 -0.20072363\n",
      "  -0.20072363]\n",
      " [-0.19631563 -0.19631563 -0.19631563 ... -0.2289701  -0.2289701\n",
      "  -0.2289701 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22380689 -0.22380689 -0.22380689 ... -0.19422352 -0.19422352\n",
      "  -0.19422352]\n",
      " [-0.24039614 -0.24039614 -0.24039614 ... -0.2300319  -0.2300319\n",
      "  -0.2300319 ]\n",
      " [-0.19883989 -0.19883989 -0.19883989 ... -0.1898595  -0.1898595\n",
      "  -0.1898595 ]\n",
      " ...\n",
      " [-0.22856504 -0.22856504 -0.22856504 ... -0.21286114 -0.21286114\n",
      "  -0.21286114]\n",
      " [-0.20020795 -0.20020795 -0.20020795 ... -0.21286114 -0.21286114\n",
      "  -0.21286114]\n",
      " [-0.19560274 -0.19560274 -0.19560274 ... -0.20738688 -0.20738688\n",
      "  -0.20738688]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.22995481 -0.22995481 -0.22995481 ... -0.21853848 -0.21853848\n",
      "  -0.21853848]\n",
      " [-0.23600498 -0.23600498 -0.23600498 ... -0.2581311  -0.2581311\n",
      "  -0.2581311 ]\n",
      " [-0.21437643 -0.21437643 -0.21437643 ... -0.2107599  -0.2107599\n",
      "  -0.2107599 ]\n",
      " ...\n",
      " [-0.23301971 -0.23301971 -0.23301971 ... -0.21959472 -0.21959472\n",
      "  -0.21959472]\n",
      " [-0.18852225 -0.18852225 -0.18852225 ... -0.24855354 -0.24855354\n",
      "  -0.24855354]\n",
      " [-0.21852644 -0.21852644 -0.21852644 ... -0.21445352 -0.21445352\n",
      "  -0.21445352]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.15020654 -0.15020654 -0.15020654 ... -0.1996899  -0.1996899\n",
      "  -0.1996899 ]\n",
      " [-0.21313298 -0.21313298 -0.21313298 ... -0.15713787 -0.15713787\n",
      "  -0.15713787]\n",
      " [-0.18417332 -0.18417332 -0.18417332 ... -0.19010715 -0.19010715\n",
      "  -0.19010715]\n",
      " ...\n",
      " [-0.19829853 -0.19829853 -0.19829853 ... -0.20461172 -0.20461172\n",
      "  -0.20461172]\n",
      " [-0.23294638 -0.23294638 -0.23294638 ... -0.20603469 -0.20603469\n",
      "  -0.20603469]\n",
      " [-0.20236556 -0.20236556 -0.20236556 ... -0.21960276 -0.21960276\n",
      "  -0.21960276]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.22681236 -0.22681236 -0.22681236 ... -0.22965033 -0.22965033\n",
      "  -0.22965033]\n",
      " [-0.18588705 -0.18588705 -0.18588705 ... -0.20343885 -0.20343885\n",
      "  -0.20343885]\n",
      " [-0.17819194 -0.17819194 -0.17819194 ... -0.23099521 -0.23099521\n",
      "  -0.23099521]\n",
      " ...\n",
      " [-0.21717061 -0.21717061 -0.21717061 ... -0.20976333 -0.20976333\n",
      "  -0.20976333]\n",
      " [-0.20695065 -0.20695065 -0.20695065 ... -0.19199948 -0.19199948\n",
      "  -0.19199948]\n",
      " [-0.19557588 -0.19557588 -0.19557588 ... -0.22023326 -0.22023326\n",
      "  -0.22023326]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.22206002 -0.22206002 -0.22206002 ... -0.19832125 -0.19832125\n",
      "  -0.19832125]\n",
      " [-0.24863064 -0.24863064 -0.24863064 ... -0.19937903 -0.19937903\n",
      "  -0.19937903]\n",
      " [-0.17752847 -0.17752847 -0.17752847 ... -0.22930494 -0.22930494\n",
      "  -0.22930494]\n",
      " ...\n",
      " [-0.22625545 -0.22625545 -0.22625545 ... -0.22233872 -0.22233872\n",
      "  -0.22233872]\n",
      " [-0.203535   -0.203535   -0.203535   ... -0.2077838  -0.2077838\n",
      "  -0.2077838 ]\n",
      " [-0.19702151 -0.19702151 -0.19702151 ... -0.2174162  -0.2174162\n",
      "  -0.2174162 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  44 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20941171 -0.20941171 -0.20941171 ... -0.22879487 -0.22879487\n",
      "  -0.22879487]\n",
      " [-0.21895784 -0.21895784 -0.21895784 ... -0.23172522 -0.23172522\n",
      "  -0.23172522]\n",
      " [-0.21666765 -0.21666765 -0.21666765 ... -0.21301958 -0.21301958\n",
      "  -0.21301958]\n",
      " ...\n",
      " [-0.21469995 -0.21469995 -0.21469995 ... -0.21231383 -0.21231383\n",
      "  -0.21231383]\n",
      " [-0.21488464 -0.21488464 -0.21488464 ... -0.12033444 -0.12033444\n",
      "  -0.12033444]\n",
      " [-0.20021415 -0.20021415 -0.20021415 ... -0.22670025 -0.22670025\n",
      "  -0.22670025]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.23484477 -0.23484477 -0.23484477 ... -0.18834177 -0.18834177\n",
      "  -0.18834177]\n",
      " [-0.21303031 -0.21303031 -0.21303031 ... -0.22244355 -0.22244355\n",
      "  -0.22244355]\n",
      " [-0.2199068  -0.2199068  -0.2199068  ... -0.22984855 -0.22984855\n",
      "  -0.22984855]\n",
      " ...\n",
      " [-0.20269692 -0.20269692 -0.20269692 ... -0.19387221 -0.19387221\n",
      "  -0.19387221]\n",
      " [-0.2457422  -0.2457422  -0.2457422  ... -0.2321785  -0.2321785\n",
      "  -0.2321785 ]\n",
      " [-0.21249722 -0.21249722 -0.21249722 ... -0.21835987 -0.21835987\n",
      "  -0.21835987]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20477578 -0.20477578 -0.20477578 ... -0.20131621 -0.20131621\n",
      "  -0.20131621]\n",
      " [-0.21696258 -0.21696258 -0.21696258 ... -0.21162002 -0.21162002\n",
      "  -0.21162002]\n",
      " [-0.19794095 -0.19794095 -0.19794095 ... -0.24794364 -0.24794364\n",
      "  -0.24794364]\n",
      " ...\n",
      " [-0.2014253  -0.2014253  -0.2014253  ... -0.22713426 -0.22713426\n",
      "  -0.22713426]\n",
      " [-0.25918347 -0.25918347 -0.25918347 ... -0.2204768  -0.2204768\n",
      "  -0.2204768 ]\n",
      " [-0.23729974 -0.23729974 -0.23729974 ... -0.19819994 -0.19819994\n",
      "  -0.19819994]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22793178 -0.22793178 -0.22793178 ... -0.19693035 -0.19693035\n",
      "  -0.19693035]\n",
      " [-0.21324302 -0.21324302 -0.21324302 ... -0.20475368 -0.20475368\n",
      "  -0.20475368]\n",
      " [-0.23934181 -0.23934181 -0.23934181 ... -0.21799585 -0.21799585\n",
      "  -0.21799585]\n",
      " ...\n",
      " [-0.21667324 -0.21667324 -0.21667324 ... -0.22324328 -0.22324328\n",
      "  -0.22324328]\n",
      " [-0.22716317 -0.22716317 -0.22716317 ... -0.20568357 -0.20568357\n",
      "  -0.20568357]\n",
      " [-0.21028031 -0.21028031 -0.21028031 ... -0.20484796 -0.20484796\n",
      "  -0.20484796]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.21377933 -0.21377933 -0.21377933 ... -0.1944617  -0.1944617\n",
      "  -0.1944617 ]\n",
      " [-0.23268884 -0.23268884 -0.23268884 ... -0.21350811 -0.21350811\n",
      "  -0.21350811]\n",
      " [-0.23319757 -0.23319757 -0.23319757 ... -0.23820482 -0.23820482\n",
      "  -0.23820482]\n",
      " ...\n",
      " [-0.23338498 -0.23338498 -0.23338498 ... -0.2355853  -0.2355853\n",
      "  -0.2355853 ]\n",
      " [-0.23046789 -0.23046789 -0.23046789 ... -0.23188935 -0.23188935\n",
      "  -0.23188935]\n",
      " [-0.21932359 -0.21932359 -0.21932359 ... -0.17818141 -0.17818141\n",
      "  -0.17818141]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.29199976 -0.29199976 -0.29199976 ... -0.20904514 -0.20904514\n",
      "  -0.20904514]\n",
      " [-0.26654977 -0.26654977 -0.26654977 ... -0.20941266 -0.20941266\n",
      "  -0.20941266]\n",
      " [-0.2476656  -0.2476656  -0.2476656  ... -0.18169025 -0.18169025\n",
      "  -0.18169025]\n",
      " ...\n",
      " [-0.24200848 -0.24200848 -0.24200848 ... -0.20591111 -0.20591111\n",
      "  -0.20591111]\n",
      " [-0.23318738 -0.23318738 -0.23318738 ... -0.14290069 -0.14290069\n",
      "  -0.14290069]\n",
      " [-0.25626582 -0.25626582 -0.25626582 ... -0.22410259 -0.22410259\n",
      "  -0.22410259]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24310099 -0.24310099 -0.24310099 ... -0.24534091 -0.24534091\n",
      "  -0.24534091]\n",
      " [-0.2177439  -0.2177439  -0.2177439  ... -0.17764364 -0.17764364\n",
      "  -0.17764364]\n",
      " [-0.23841548 -0.23841548 -0.23841548 ... -0.26084432 -0.26084432\n",
      "  -0.26084432]\n",
      " ...\n",
      " [-0.2003098  -0.2003098  -0.2003098  ... -0.21229842 -0.21229842\n",
      "  -0.21229842]\n",
      " [-0.22900209 -0.22900209 -0.22900209 ... -0.19947428 -0.19947428\n",
      "  -0.19947428]\n",
      " [-0.21501103 -0.21501103 -0.21501103 ... -0.19629851 -0.19629851\n",
      "  -0.19629851]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.20372462 -0.20372462 -0.20372462 ... -0.2149935  -0.2149935\n",
      "  -0.2149935 ]\n",
      " [-0.22161953 -0.22161953 -0.22161953 ... -0.21577418 -0.21577418\n",
      "  -0.21577418]\n",
      " [-0.22017705 -0.22017705 -0.22017705 ... -0.23890266 -0.23890266\n",
      "  -0.23890266]\n",
      " ...\n",
      " [-0.20077053 -0.20077053 -0.20077053 ... -0.24974377 -0.24974377\n",
      "  -0.24974377]\n",
      " [-0.22918917 -0.22918917 -0.22918917 ... -0.2074714  -0.2074714\n",
      "  -0.2074714 ]\n",
      " [-0.20116279 -0.20116279 -0.20116279 ... -0.22309421 -0.22309421\n",
      "  -0.22309421]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  45 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.21064876 -0.21064876 -0.21064876 ... -0.24254398 -0.24254398\n",
      "  -0.24254398]\n",
      " [-0.263172   -0.263172   -0.263172   ... -0.2429046  -0.2429046\n",
      "  -0.2429046 ]\n",
      " [-0.22489491 -0.22489491 -0.22489491 ... -0.22809309 -0.22809309\n",
      "  -0.22809309]\n",
      " ...\n",
      " [-0.20699756 -0.20699756 -0.20699756 ... -0.23251623 -0.23251623\n",
      "  -0.23251623]\n",
      " [-0.2523109  -0.2523109  -0.2523109  ... -0.24883932 -0.24883932\n",
      "  -0.24883932]\n",
      " [-0.2352324  -0.2352324  -0.2352324  ... -0.23405111 -0.23405111\n",
      "  -0.23405111]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.18547654 -0.18547654 -0.18547654 ... -0.25082597 -0.25082597\n",
      "  -0.25082597]\n",
      " [-0.21596672 -0.21596672 -0.21596672 ... -0.20203881 -0.20203881\n",
      "  -0.20203881]\n",
      " [-0.23829542 -0.23829542 -0.23829542 ... -0.24749565 -0.24749565\n",
      "  -0.24749565]\n",
      " ...\n",
      " [-0.19318958 -0.19318958 -0.19318958 ... -0.2439415  -0.2439415\n",
      "  -0.2439415 ]\n",
      " [-0.19603108 -0.19603108 -0.19603108 ... -0.22600986 -0.22600986\n",
      "  -0.22600986]\n",
      " [-0.20917259 -0.20917259 -0.20917259 ... -0.23945594 -0.23945594\n",
      "  -0.23945594]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.22262391 -0.22262391 -0.22262391 ... -0.22860101 -0.22860101\n",
      "  -0.22860101]\n",
      " [-0.20186876 -0.20186876 -0.20186876 ... -0.2260919  -0.2260919\n",
      "  -0.2260919 ]\n",
      " [-0.28081435 -0.28081435 -0.28081435 ... -0.23695879 -0.23695879\n",
      "  -0.23695879]\n",
      " ...\n",
      " [-0.25184095 -0.25184095 -0.25184095 ... -0.21837062 -0.21837062\n",
      "  -0.21837062]\n",
      " [-0.24862877 -0.24862877 -0.24862877 ... -0.17538902 -0.17538902\n",
      "  -0.17538902]\n",
      " [-0.22637233 -0.22637233 -0.22637233 ... -0.2417508  -0.2417508\n",
      "  -0.2417508 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.19883066 -0.19883066 -0.19883066 ... -0.19622585 -0.19622585\n",
      "  -0.19622585]\n",
      " [-0.21925583 -0.21925583 -0.21925583 ... -0.23261079 -0.23261079\n",
      "  -0.23261079]\n",
      " [-0.18847016 -0.18847016 -0.18847016 ... -0.26633853 -0.26633853\n",
      "  -0.26633853]\n",
      " ...\n",
      " [-0.18821886 -0.18821886 -0.18821886 ... -0.27155423 -0.27155423\n",
      "  -0.27155423]\n",
      " [-0.2501466  -0.2501466  -0.2501466  ... -0.22002529 -0.22002529\n",
      "  -0.22002529]\n",
      " [-0.24444342 -0.24444342 -0.24444342 ... -0.25324902 -0.25324902\n",
      "  -0.25324902]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.23610817 -0.23610817 -0.23610817 ... -0.22414704 -0.22414704\n",
      "  -0.22414704]\n",
      " [-0.22688864 -0.22688864 -0.22688864 ... -0.2293413  -0.2293413\n",
      "  -0.2293413 ]\n",
      " [-0.24085163 -0.24085163 -0.24085163 ... -0.20400323 -0.20400323\n",
      "  -0.20400323]\n",
      " ...\n",
      " [-0.22667515 -0.22667515 -0.22667515 ... -0.23965636 -0.23965636\n",
      "  -0.23965636]\n",
      " [-0.22541106 -0.22541106 -0.22541106 ... -0.24549489 -0.24549489\n",
      "  -0.24549489]\n",
      " [-0.23782432 -0.23782432 -0.23782432 ... -0.21384251 -0.21384251\n",
      "  -0.21384251]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.29475516 -0.29475516 -0.29475516 ... -0.23113188 -0.23113188\n",
      "  -0.23113188]\n",
      " [-0.20836423 -0.20836423 -0.20836423 ... -0.2909909  -0.2909909\n",
      "  -0.2909909 ]\n",
      " [-0.24533299 -0.24533299 -0.24533299 ... -0.23454939 -0.23454939\n",
      "  -0.23454939]\n",
      " ...\n",
      " [-0.223365   -0.223365   -0.223365   ... -0.2329693  -0.2329693\n",
      "  -0.2329693 ]\n",
      " [-0.2190069  -0.2190069  -0.2190069  ... -0.18195356 -0.18195356\n",
      "  -0.18195356]\n",
      " [-0.20287508 -0.20287508 -0.20287508 ... -0.23335996 -0.23335996\n",
      "  -0.23335996]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.16991316 -0.16991316 -0.16991316 ... -0.2120316  -0.2120316\n",
      "  -0.2120316 ]\n",
      " [-0.24349834 -0.24349834 -0.24349834 ... -0.21969229 -0.21969229\n",
      "  -0.21969229]\n",
      " [-0.21037531 -0.21037531 -0.21037531 ... -0.28118518 -0.28118518\n",
      "  -0.28118518]\n",
      " ...\n",
      " [-0.21505865 -0.21505865 -0.21505865 ... -0.2511947  -0.2511947\n",
      "  -0.2511947 ]\n",
      " [-0.23528098 -0.23528098 -0.23528098 ... -0.22381006 -0.22381006\n",
      "  -0.22381006]\n",
      " [-0.22421046 -0.22421046 -0.22421046 ... -0.21078803 -0.21078803\n",
      "  -0.21078803]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.24045017 -0.24045017 -0.24045017 ... -0.21728046 -0.21728046\n",
      "  -0.21728046]\n",
      " [-0.21068266 -0.21068266 -0.21068266 ... -0.29102466 -0.29102466\n",
      "  -0.29102466]\n",
      " [-0.25229686 -0.25229686 -0.25229686 ... -0.22505    -0.22505\n",
      "  -0.22505   ]\n",
      " ...\n",
      " [-0.24730127 -0.24730127 -0.24730127 ... -0.20688093 -0.20688093\n",
      "  -0.20688093]\n",
      " [-0.25170583 -0.25170583 -0.25170583 ... -0.2142938  -0.2142938\n",
      "  -0.2142938 ]\n",
      " [-0.20403975 -0.20403975 -0.20403975 ... -0.23615748 -0.23615748\n",
      "  -0.23615748]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  46 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23491383 -0.23491383 -0.23491383 ... -0.24627867 -0.24627867\n",
      "  -0.24627867]\n",
      " [-0.23250745 -0.23250745 -0.23250745 ... -0.21174543 -0.21174543\n",
      "  -0.21174543]\n",
      " [-0.20785967 -0.20785967 -0.20785967 ... -0.2619608  -0.2619608\n",
      "  -0.2619608 ]\n",
      " ...\n",
      " [-0.24293315 -0.24293315 -0.24293315 ... -0.25826    -0.25826\n",
      "  -0.25826   ]\n",
      " [-0.23931171 -0.23931171 -0.23931171 ... -0.20098595 -0.20098595\n",
      "  -0.20098595]\n",
      " [-0.22243682 -0.22243682 -0.22243682 ... -0.24403623 -0.24403623\n",
      "  -0.24403623]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.2253752  -0.2253752  -0.2253752  ... -0.23098066 -0.23098066\n",
      "  -0.23098066]\n",
      " [-0.2145882  -0.2145882  -0.2145882  ... -0.22161697 -0.22161697\n",
      "  -0.22161697]\n",
      " [-0.24860863 -0.24860863 -0.24860863 ... -0.23111928 -0.23111928\n",
      "  -0.23111928]\n",
      " ...\n",
      " [-0.22765952 -0.22765952 -0.22765952 ... -0.24573767 -0.24573767\n",
      "  -0.24573767]\n",
      " [-0.26042563 -0.26042563 -0.26042563 ... -0.26255137 -0.26255137\n",
      "  -0.26255137]\n",
      " [-0.21613336 -0.21613336 -0.21613336 ... -0.20065142 -0.20065142\n",
      "  -0.20065142]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23944479 -0.23944479 -0.23944479 ... -0.22587812 -0.22587812\n",
      "  -0.22587812]\n",
      " [-0.21782374 -0.21782374 -0.21782374 ... -0.26047242 -0.26047242\n",
      "  -0.26047242]\n",
      " [-0.26564544 -0.26564544 -0.26564544 ... -0.22079058 -0.22079058\n",
      "  -0.22079058]\n",
      " ...\n",
      " [-0.23171042 -0.23171042 -0.23171042 ... -0.25279295 -0.25279295\n",
      "  -0.25279295]\n",
      " [-0.26097968 -0.26097968 -0.26097968 ... -0.26589334 -0.26589334\n",
      "  -0.26589334]\n",
      " [-0.24931204 -0.24931204 -0.24931204 ... -0.23520908 -0.23520908\n",
      "  -0.23520908]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.16267727 -0.16267727 -0.16267727 ... -0.2507499  -0.2507499\n",
      "  -0.2507499 ]\n",
      " [-0.26257366 -0.26257366 -0.26257366 ... -0.21858224 -0.21858224\n",
      "  -0.21858224]\n",
      " [-0.25656712 -0.25656712 -0.25656712 ... -0.242037   -0.242037\n",
      "  -0.242037  ]\n",
      " ...\n",
      " [-0.21726367 -0.21726367 -0.21726367 ... -0.22871429 -0.22871429\n",
      "  -0.22871429]\n",
      " [-0.26604912 -0.26604912 -0.26604912 ... -0.21447735 -0.21447735\n",
      "  -0.21447735]\n",
      " [-0.28063473 -0.28063473 -0.28063473 ... -0.27228034 -0.27228034\n",
      "  -0.27228034]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.24257153 -0.24257153 -0.24257153 ... -0.26337427 -0.26337427\n",
      "  -0.26337427]\n",
      " [-0.2270751  -0.2270751  -0.2270751  ... -0.2472919  -0.2472919\n",
      "  -0.2472919 ]\n",
      " [-0.18585563 -0.18585563 -0.18585563 ... -0.21454291 -0.21454291\n",
      "  -0.21454291]\n",
      " ...\n",
      " [-0.23089187 -0.23089187 -0.23089187 ... -0.22075233 -0.22075233\n",
      "  -0.22075233]\n",
      " [-0.18494785 -0.18494785 -0.18494785 ... -0.204349   -0.204349\n",
      "  -0.204349  ]\n",
      " [-0.2327629  -0.2327629  -0.2327629  ... -0.22004415 -0.22004415\n",
      "  -0.22004415]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.22213867 -0.22213867 -0.22213867 ... -0.22301687 -0.22301687\n",
      "  -0.22301687]\n",
      " [-0.2203238  -0.2203238  -0.2203238  ... -0.25456634 -0.25456634\n",
      "  -0.25456634]\n",
      " [-0.21628502 -0.21628502 -0.21628502 ... -0.21706459 -0.21706459\n",
      "  -0.21706459]\n",
      " ...\n",
      " [-0.19768375 -0.19768375 -0.19768375 ... -0.24669173 -0.24669173\n",
      "  -0.24669173]\n",
      " [-0.21372736 -0.21372736 -0.21372736 ... -0.24038377 -0.24038377\n",
      "  -0.24038377]\n",
      " [-0.23148082 -0.23148082 -0.23148082 ... -0.25322795 -0.25322795\n",
      "  -0.25322795]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.23052657 -0.23052657 -0.23052657 ... -0.23040502 -0.23040502\n",
      "  -0.23040502]\n",
      " [-0.23591067 -0.23591067 -0.23591067 ... -0.28550833 -0.28550833\n",
      "  -0.28550833]\n",
      " [-0.23456453 -0.23456453 -0.23456453 ... -0.20995072 -0.20995072\n",
      "  -0.20995072]\n",
      " ...\n",
      " [-0.25757095 -0.25757095 -0.25757095 ... -0.25158104 -0.25158104\n",
      "  -0.25158104]\n",
      " [-0.24772446 -0.24772446 -0.24772446 ... -0.24800164 -0.24800164\n",
      "  -0.24800164]\n",
      " [-0.21180113 -0.21180113 -0.21180113 ... -0.22045311 -0.22045311\n",
      "  -0.22045311]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.23658095 -0.23658095 -0.23658095 ... -0.24895078 -0.24895078\n",
      "  -0.24895078]\n",
      " [-0.2762044  -0.2762044  -0.2762044  ... -0.22036788 -0.22036788\n",
      "  -0.22036788]\n",
      " [-0.23890695 -0.23890695 -0.23890695 ... -0.22361049 -0.22361049\n",
      "  -0.22361049]\n",
      " ...\n",
      " [-0.26688528 -0.26688528 -0.26688528 ... -0.22311786 -0.22311786\n",
      "  -0.22311786]\n",
      " [-0.24777064 -0.24777064 -0.24777064 ... -0.25046033 -0.25046033\n",
      "  -0.25046033]\n",
      " [-0.24965268 -0.24965268 -0.24965268 ... -0.24933283 -0.24933283\n",
      "  -0.24933283]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  47 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23451921 -0.23451921 -0.23451921 ... -0.21214193 -0.21214193\n",
      "  -0.21214193]\n",
      " [-0.25325713 -0.25325713 -0.25325713 ... -0.21795976 -0.21795976\n",
      "  -0.21795976]\n",
      " [-0.25333247 -0.25333247 -0.25333247 ... -0.22834992 -0.22834992\n",
      "  -0.22834992]\n",
      " ...\n",
      " [-0.23165695 -0.23165695 -0.23165695 ... -0.28749442 -0.28749442\n",
      "  -0.28749442]\n",
      " [-0.26500463 -0.26500463 -0.26500463 ... -0.23177871 -0.23177871\n",
      "  -0.23177871]\n",
      " [-0.2440828  -0.2440828  -0.2440828  ... -0.22278678 -0.22278678\n",
      "  -0.22278678]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24590527 -0.24590527 -0.24590527 ... -0.24880289 -0.24880289\n",
      "  -0.24880289]\n",
      " [-0.26372167 -0.26372167 -0.26372167 ... -0.2242352  -0.2242352\n",
      "  -0.2242352 ]\n",
      " [-0.24221244 -0.24221244 -0.24221244 ... -0.22120337 -0.22120337\n",
      "  -0.22120337]\n",
      " ...\n",
      " [-0.21737571 -0.21737571 -0.21737571 ... -0.2782239  -0.2782239\n",
      "  -0.2782239 ]\n",
      " [-0.21968947 -0.21968947 -0.21968947 ... -0.24862336 -0.24862336\n",
      "  -0.24862336]\n",
      " [-0.256257   -0.256257   -0.256257   ... -0.20575374 -0.20575374\n",
      "  -0.20575374]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.25709054 -0.25709054 -0.25709054 ... -0.2584923  -0.2584923\n",
      "  -0.2584923 ]\n",
      " [-0.18645743 -0.18645743 -0.18645743 ... -0.21831413 -0.21831413\n",
      "  -0.21831413]\n",
      " [-0.24047865 -0.24047865 -0.24047865 ... -0.21677537 -0.21677537\n",
      "  -0.21677537]\n",
      " ...\n",
      " [-0.25742877 -0.25742877 -0.25742877 ... -0.24074368 -0.24074368\n",
      "  -0.24074368]\n",
      " [-0.25899854 -0.25899854 -0.25899854 ... -0.23618649 -0.23618649\n",
      "  -0.23618649]\n",
      " [-0.19443068 -0.19443068 -0.19443068 ... -0.22870752 -0.22870752\n",
      "  -0.22870752]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.28076464 -0.28076464 -0.28076464 ... -0.2286093  -0.2286093\n",
      "  -0.2286093 ]\n",
      " [-0.22625977 -0.22625977 -0.22625977 ... -0.26011246 -0.26011246\n",
      "  -0.26011246]\n",
      " [-0.18803981 -0.18803981 -0.18803981 ... -0.24777994 -0.24777994\n",
      "  -0.24777994]\n",
      " ...\n",
      " [-0.2296507  -0.2296507  -0.2296507  ... -0.2661998  -0.2661998\n",
      "  -0.2661998 ]\n",
      " [-0.19193313 -0.19193313 -0.19193313 ... -0.2472641  -0.2472641\n",
      "  -0.2472641 ]\n",
      " [-0.33080727 -0.33080727 -0.33080727 ... -0.2733827  -0.2733827\n",
      "  -0.2733827 ]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.25355703 -0.25355703 -0.25355703 ... -0.24091485 -0.24091485\n",
      "  -0.24091485]\n",
      " [-0.20549758 -0.20549758 -0.20549758 ... -0.24325669 -0.24325669\n",
      "  -0.24325669]\n",
      " [-0.22640352 -0.22640352 -0.22640352 ... -0.26144394 -0.26144394\n",
      "  -0.26144394]\n",
      " ...\n",
      " [-0.23341356 -0.23341356 -0.23341356 ... -0.21855265 -0.21855265\n",
      "  -0.21855265]\n",
      " [-0.2104574  -0.2104574  -0.2104574  ... -0.24331464 -0.24331464\n",
      "  -0.24331464]\n",
      " [-0.235336   -0.235336   -0.235336   ... -0.2674439  -0.2674439\n",
      "  -0.2674439 ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.3204193  -0.3204193  -0.3204193  ... -0.24860638 -0.24860638\n",
      "  -0.24860638]\n",
      " [-0.24970536 -0.24970536 -0.24970536 ... -0.25460953 -0.25460953\n",
      "  -0.25460953]\n",
      " [-0.27536196 -0.27536196 -0.27536196 ... -0.24871038 -0.24871038\n",
      "  -0.24871038]\n",
      " ...\n",
      " [-0.2323427  -0.2323427  -0.2323427  ... -0.27511564 -0.27511564\n",
      "  -0.27511564]\n",
      " [-0.23573714 -0.23573714 -0.23573714 ... -0.23997076 -0.23997076\n",
      "  -0.23997076]\n",
      " [-0.22872487 -0.22872487 -0.22872487 ... -0.22327095 -0.22327095\n",
      "  -0.22327095]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24487048 -0.24487048 -0.24487048 ... -0.23154595 -0.23154595\n",
      "  -0.23154595]\n",
      " [-0.22615515 -0.22615515 -0.22615515 ... -0.28297108 -0.28297108\n",
      "  -0.28297108]\n",
      " [-0.2520145  -0.2520145  -0.2520145  ... -0.22750205 -0.22750205\n",
      "  -0.22750205]\n",
      " ...\n",
      " [-0.25898013 -0.25898013 -0.25898013 ... -0.1769661  -0.1769661\n",
      "  -0.1769661 ]\n",
      " [-0.21465892 -0.21465892 -0.21465892 ... -0.24945356 -0.24945356\n",
      "  -0.24945356]\n",
      " [-0.24992162 -0.24992162 -0.24992162 ... -0.25778937 -0.25778937\n",
      "  -0.25778937]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.24258155 -0.24258155 -0.24258155 ... -0.23708788 -0.23708788\n",
      "  -0.23708788]\n",
      " [-0.23868093 -0.23868093 -0.23868093 ... -0.23668614 -0.23668614\n",
      "  -0.23668614]\n",
      " [-0.24349953 -0.24349953 -0.24349953 ... -0.23876306 -0.23876306\n",
      "  -0.23876306]\n",
      " ...\n",
      " [-0.20976569 -0.20976569 -0.20976569 ... -0.26134673 -0.26134673\n",
      "  -0.26134673]\n",
      " [-0.22285709 -0.22285709 -0.22285709 ... -0.26490694 -0.26490694\n",
      "  -0.26490694]\n",
      " [-0.25237203 -0.25237203 -0.25237203 ... -0.23408422 -0.23408422\n",
      "  -0.23408422]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ step  48 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.24711709 -0.24711709 -0.24711709 ... -0.28307924 -0.28307924\n",
      "  -0.28307924]\n",
      " [-0.2479815  -0.2479815  -0.2479815  ... -0.2596137  -0.2596137\n",
      "  -0.2596137 ]\n",
      " [-0.23282254 -0.23282254 -0.23282254 ... -0.24117479 -0.24117479\n",
      "  -0.24117479]\n",
      " ...\n",
      " [-0.27805972 -0.27805972 -0.27805972 ... -0.24752554 -0.24752554\n",
      "  -0.24752554]\n",
      " [-0.2415568  -0.2415568  -0.2415568  ... -0.2516385  -0.2516385\n",
      "  -0.2516385 ]\n",
      " [-0.2403656  -0.2403656  -0.2403656  ... -0.24829827 -0.24829827\n",
      "  -0.24829827]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24093132 -0.24093132 -0.24093132 ... -0.2616806  -0.2616806\n",
      "  -0.2616806 ]\n",
      " [-0.2341744  -0.2341744  -0.2341744  ... -0.23874548 -0.23874548\n",
      "  -0.23874548]\n",
      " [-0.23052762 -0.23052762 -0.23052762 ... -0.23824754 -0.23824754\n",
      "  -0.23824754]\n",
      " ...\n",
      " [-0.2503054  -0.2503054  -0.2503054  ... -0.2665841  -0.2665841\n",
      "  -0.2665841 ]\n",
      " [-0.25577426 -0.25577426 -0.25577426 ... -0.23218544 -0.23218544\n",
      "  -0.23218544]\n",
      " [-0.22642556 -0.22642556 -0.22642556 ... -0.19263582 -0.19263582\n",
      "  -0.19263582]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24006249 -0.24006249 -0.24006249 ... -0.19978298 -0.19978298\n",
      "  -0.19978298]\n",
      " [-0.19832012 -0.19832012 -0.19832012 ... -0.2785647  -0.2785647\n",
      "  -0.2785647 ]\n",
      " [-0.24075241 -0.24075241 -0.24075241 ... -0.26310307 -0.26310307\n",
      "  -0.26310307]\n",
      " ...\n",
      " [-0.23475087 -0.23475087 -0.23475087 ... -0.12876263 -0.12876263\n",
      "  -0.12876263]\n",
      " [-0.24222331 -0.24222331 -0.24222331 ... -0.2131702  -0.2131702\n",
      "  -0.2131702 ]\n",
      " [-0.28012648 -0.28012648 -0.28012648 ... -0.20706888 -0.20706888\n",
      "  -0.20706888]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2357806  -0.2357806  -0.2357806  ... -0.22080818 -0.22080818\n",
      "  -0.22080818]\n",
      " [-0.23020133 -0.23020133 -0.23020133 ... -0.23741256 -0.23741256\n",
      "  -0.23741256]\n",
      " [-0.2010029  -0.2010029  -0.2010029  ... -0.24844693 -0.24844693\n",
      "  -0.24844693]\n",
      " ...\n",
      " [-0.15202963 -0.15202963 -0.15202963 ... -0.26789013 -0.26789013\n",
      "  -0.26789013]\n",
      " [-0.24009094 -0.24009094 -0.24009094 ... -0.25889638 -0.25889638\n",
      "  -0.25889638]\n",
      " [-0.21766588 -0.21766588 -0.21766588 ... -0.27945027 -0.27945027\n",
      "  -0.27945027]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.22601026 -0.22601026 -0.22601026 ... -0.2649148  -0.2649148\n",
      "  -0.2649148 ]\n",
      " [-0.24677181 -0.24677181 -0.24677181 ... -0.22742358 -0.22742358\n",
      "  -0.22742358]\n",
      " [-0.2623896  -0.2623896  -0.2623896  ... -0.22985499 -0.22985499\n",
      "  -0.22985499]\n",
      " ...\n",
      " [-0.275735   -0.275735   -0.275735   ... -0.2572251  -0.2572251\n",
      "  -0.2572251 ]\n",
      " [-0.2673136  -0.2673136  -0.2673136  ... -0.29593012 -0.29593012\n",
      "  -0.29593012]\n",
      " [-0.21580365 -0.21580365 -0.21580365 ... -0.24056146 -0.24056146\n",
      "  -0.24056146]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.23972781 -0.23972781 -0.23972781 ... -0.26600868 -0.26600868\n",
      "  -0.26600868]\n",
      " [-0.21774828 -0.21774828 -0.21774828 ... -0.22890219 -0.22890219\n",
      "  -0.22890219]\n",
      " [-0.23518531 -0.23518531 -0.23518531 ... -0.2996247  -0.2996247\n",
      "  -0.2996247 ]\n",
      " ...\n",
      " [-0.25669935 -0.25669935 -0.25669935 ... -0.26554105 -0.26554105\n",
      "  -0.26554105]\n",
      " [-0.24645734 -0.24645734 -0.24645734 ... -0.2464787  -0.2464787\n",
      "  -0.2464787 ]\n",
      " [-0.22436646 -0.22436646 -0.22436646 ... -0.19629312 -0.19629312\n",
      "  -0.19629312]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.24281007 -0.24281007 -0.24281007 ... -0.22957414 -0.22957414\n",
      "  -0.22957414]\n",
      " [-0.26333562 -0.26333562 -0.26333562 ... -0.23378037 -0.23378037\n",
      "  -0.23378037]\n",
      " [-0.26403242 -0.26403242 -0.26403242 ... -0.2572427  -0.2572427\n",
      "  -0.2572427 ]\n",
      " ...\n",
      " [-0.25578192 -0.25578192 -0.25578192 ... -0.25533107 -0.25533107\n",
      "  -0.25533107]\n",
      " [-0.27287054 -0.27287054 -0.27287054 ... -0.21852249 -0.21852249\n",
      "  -0.21852249]\n",
      " [-0.27295098 -0.27295098 -0.27295098 ... -0.21705586 -0.21705586\n",
      "  -0.21705586]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.24725077 -0.24725077 -0.24725077 ... -0.27231422 -0.27231422\n",
      "  -0.27231422]\n",
      " [-0.23564512 -0.23564512 -0.23564512 ... -0.2074919  -0.2074919\n",
      "  -0.2074919 ]\n",
      " [-0.21403398 -0.21403398 -0.21403398 ... -0.30093634 -0.30093634\n",
      "  -0.30093634]\n",
      " ...\n",
      " [-0.23466569 -0.23466569 -0.23466569 ... -0.21984595 -0.21984595\n",
      "  -0.21984595]\n",
      " [-0.23673092 -0.23673092 -0.23673092 ... -0.22425222 -0.22425222\n",
      "  -0.22425222]\n",
      " [-0.20856981 -0.20856981 -0.20856981 ... -0.26559356 -0.26559356\n",
      "  -0.26559356]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------ step  49 ------------------------------\n",
      "[INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20030972 -0.20030972 -0.20030972 ... -0.2455024  -0.2455024\n",
      "  -0.2455024 ]\n",
      " [-0.22935075 -0.22935075 -0.22935075 ... -0.28199342 -0.28199342\n",
      "  -0.28199342]\n",
      " [-0.25484663 -0.25484663 -0.25484663 ... -0.24498652 -0.24498652\n",
      "  -0.24498652]\n",
      " ...\n",
      " [-0.2420542  -0.2420542  -0.2420542  ... -0.25748128 -0.25748128\n",
      "  -0.25748128]\n",
      " [-0.24584286 -0.24584286 -0.24584286 ... -0.26521006 -0.26521006\n",
      "  -0.26521006]\n",
      " [-0.23275915 -0.23275915 -0.23275915 ... -0.29811582 -0.29811582\n",
      "  -0.29811582]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24068819 -0.24068819 -0.24068819 ... -0.24306706 -0.24306706\n",
      "  -0.24306706]\n",
      " [-0.23070751 -0.23070751 -0.23070751 ... -0.26770514 -0.26770514\n",
      "  -0.26770514]\n",
      " [-0.23909771 -0.23909771 -0.23909771 ... -0.2282634  -0.2282634\n",
      "  -0.2282634 ]\n",
      " ...\n",
      " [-0.21236053 -0.21236053 -0.21236053 ... -0.23154336 -0.23154336\n",
      "  -0.23154336]\n",
      " [-0.21440458 -0.21440458 -0.21440458 ... -0.22316912 -0.22316912\n",
      "  -0.22316912]\n",
      " [-0.32416934 -0.32416934 -0.32416934 ... -0.29144797 -0.29144797\n",
      "  -0.29144797]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2525183  -0.2525183  -0.2525183  ... -0.21462378 -0.21462378\n",
      "  -0.21462378]\n",
      " [-0.24385303 -0.24385303 -0.24385303 ... -0.24511114 -0.24511114\n",
      "  -0.24511114]\n",
      " [-0.22818327 -0.22818327 -0.22818327 ... -0.2691502  -0.2691502\n",
      "  -0.2691502 ]\n",
      " ...\n",
      " [-0.23801959 -0.23801959 -0.23801959 ... -0.24068475 -0.24068475\n",
      "  -0.24068475]\n",
      " [-0.26306295 -0.26306295 -0.26306295 ... -0.1888154  -0.1888154\n",
      "  -0.1888154 ]\n",
      " [-0.23668343 -0.23668343 -0.23668343 ... -0.25793946 -0.25793946\n",
      "  -0.25793946]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2516389  -0.2516389  -0.2516389  ... -0.23199034 -0.23199034\n",
      "  -0.23199034]\n",
      " [-0.223272   -0.223272   -0.223272   ... -0.21958078 -0.21958078\n",
      "  -0.21958078]\n",
      " [-0.25457892 -0.25457892 -0.25457892 ... -0.20346126 -0.20346126\n",
      "  -0.20346126]\n",
      " ...\n",
      " [-0.2416003  -0.2416003  -0.2416003  ... -0.2650041  -0.2650041\n",
      "  -0.2650041 ]\n",
      " [-0.26054987 -0.26054987 -0.26054987 ... -0.24629539 -0.24629539\n",
      "  -0.24629539]\n",
      " [-0.24619427 -0.24619427 -0.24619427 ... -0.25869498 -0.25869498\n",
      "  -0.25869498]], shape=(8192, 40), dtype=float32),\n",
      "  4: tf.Tensor(\n",
      "[[-0.22218853 -0.22218853 -0.22218853 ... -0.24335687 -0.24335687\n",
      "  -0.24335687]\n",
      " [-0.26132077 -0.26132077 -0.26132077 ... -0.20188732 -0.20188732\n",
      "  -0.20188732]\n",
      " [-0.23360708 -0.23360708 -0.23360708 ... -0.25797567 -0.25797567\n",
      "  -0.25797567]\n",
      " ...\n",
      " [-0.26767823 -0.26767823 -0.26767823 ... -0.2317605  -0.2317605\n",
      "  -0.2317605 ]\n",
      " [-0.2141102  -0.2141102  -0.2141102  ... -0.2361359  -0.2361359\n",
      "  -0.2361359 ]\n",
      " [-0.24454813 -0.24454813 -0.24454813 ... -0.223708   -0.223708\n",
      "  -0.223708  ]], shape=(8192, 40), dtype=float32),\n",
      "  5: tf.Tensor(\n",
      "[[-0.23050715 -0.23050715 -0.23050715 ... -0.25757068 -0.25757068\n",
      "  -0.25757068]\n",
      " [-0.24026765 -0.24026765 -0.24026765 ... -0.22529416 -0.22529416\n",
      "  -0.22529416]\n",
      " [-0.23113976 -0.23113976 -0.23113976 ... -0.25137943 -0.25137943\n",
      "  -0.25137943]\n",
      " ...\n",
      " [-0.23757485 -0.23757485 -0.23757485 ... -0.23921582 -0.23921582\n",
      "  -0.23921582]\n",
      " [-0.21629287 -0.21629287 -0.21629287 ... -0.23169547 -0.23169547\n",
      "  -0.23169547]\n",
      " [-0.2266556  -0.2266556  -0.2266556  ... -0.24863185 -0.24863185\n",
      "  -0.24863185]], shape=(8192, 40), dtype=float32),\n",
      "  6: tf.Tensor(\n",
      "[[-0.21322648 -0.21322648 -0.21322648 ... -0.24383168 -0.24383168\n",
      "  -0.24383168]\n",
      " [-0.26300365 -0.26300365 -0.26300365 ... -0.269572   -0.269572\n",
      "  -0.269572  ]\n",
      " [-0.24477741 -0.24477741 -0.24477741 ... -0.2153987  -0.2153987\n",
      "  -0.2153987 ]\n",
      " ...\n",
      " [-0.2048559  -0.2048559  -0.2048559  ... -0.2691518  -0.2691518\n",
      "  -0.2691518 ]\n",
      " [-0.24257539 -0.24257539 -0.24257539 ... -0.24507838 -0.24507838\n",
      "  -0.24507838]\n",
      " [-0.23949063 -0.23949063 -0.23949063 ... -0.24337088 -0.24337088\n",
      "  -0.24337088]], shape=(8192, 40), dtype=float32),\n",
      "  7: tf.Tensor(\n",
      "[[-0.25338995 -0.25338995 -0.25338995 ... -0.26412585 -0.26412585\n",
      "  -0.26412585]\n",
      " [-0.22636439 -0.22636439 -0.22636439 ... -0.20998524 -0.20998524\n",
      "  -0.20998524]\n",
      " [-0.23050715 -0.23050715 -0.23050715 ... -0.20203522 -0.20203522\n",
      "  -0.20203522]\n",
      " ...\n",
      " [-0.31743342 -0.31743342 -0.31743342 ... -0.25191787 -0.25191787\n",
      "  -0.25191787]\n",
      " [-0.16408359 -0.16408359 -0.16408359 ... -0.23854637 -0.23854637\n",
      "  -0.23854637]\n",
      " [-0.2500958  -0.2500958  -0.2500958  ... -0.24296874 -0.24296874\n",
      "  -0.24296874]], shape=(8192, 40), dtype=float32)\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# train SOK Demo Model, this command will print each iteration's embedding vector \n",
    "sok_results = test_sok_demo(args, init_tensors, *random_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e44161",
   "metadata": {},
   "source": [
    "Finally, check the consistency of the embedding vectors obtained from TensorFlow ans SOK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2ad80e6a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[INFO]: With MirroredStrategy, when 8 GPUs are used, the embedding vectors obtained from TensorFlowand SOK are consistent for 50 iterations\n"
     ]
    }
   ],
   "source": [
    "if (len(sok_results) != len(tf_results)):\n",
    "    raise ValueError(\"The length of sok results is not equal to that of TensorFlow.\")\n",
    "if (len(tf_results) != args[\"iter_num\"]):\n",
    "    raise ValueError(\"The length of embedding vectors: %d is not equal to iteration number: %d.\"\n",
    "                    %(len(tf_results), args[\"iter_num\"]))\n",
    "    \n",
    "for i, sok_vector in enumerate(sok_results):\n",
    "    if args[\"gpu_num\"] != 1:\n",
    "        sok_vector = tf.stack(sok_vector.values, axis=0)\n",
    "    tf.debugging.assert_near(tf.reshape(sok_vector,\n",
    "                                        shape=[-1, tf.shape(sok_vector)[-1]]),\n",
    "                             tf_results[i],\n",
    "                             atol=1e-4,\n",
    "                             rtol=1e-4,\n",
    "                             message=\"The embedding vectors obtained from TF and SOK vary in iteration: %d\" %i)\n",
    "    \n",
    "# if no exception, then the embedding vectors for all iterations are consistent.\n",
    "print((\"\\n[INFO]: With MirroredStrategy, when %d GPUs are used, the embedding vectors obtained from TensorFlow\" \n",
    "       \"and SOK are consistent for %d iterations\") \n",
    "      %(args[\"gpu_num\"], args[\"iter_num\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90e6fbd8",
   "metadata": {},
   "source": [
    "### Multi-node, Multi-GPUs synchronized training ###"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22545715",
   "metadata": {},
   "source": [
    "**The jupyter notebook kernel must be restarted!!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53d4507",
   "metadata": {},
   "source": [
    "Firstly, specify hyper parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ddc77e54",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reset -f\n",
    "\n",
    "args = dict()\n",
    "\n",
    "args[\"iter_num\"] = 50                             # the number of training iteration\n",
    "args[\"max_vocabulary_size_per_gpu\"] = 1024\n",
    "args[\"slot_num\"] = 10                             # the number of feature fields in this embedding layer\n",
    "args[\"max_nnz\"] = 4                               # the maximum number of valid features in each slot\n",
    "args[\"embedding_vec_size\"] = 4                    # the dimension of embedding vectors\n",
    "args[\"combiner\"] = \"mean\"                         # the reduction combiner used intra slots, it can be [mean, sum]\n",
    "args[\"global_batch_size\"] = 65536                 # the globally batchsize for all GPUs\n",
    "args[\"optimizer\"] = \"plugin_adam\"                 # the optimizer used for training, it can be [plugin_adam, adam, sgd]\n",
    "args[\"ips\"] = [\"localhost\", \"localhost\"]          # specify the ip addr of each node. Here we use different GPUs to \n",
    "                                                  # simulate multi-node with single-node    \n",
    "args[\"worker_num\"] = len(args[\"ips\"])             # the number of workers in synchronized training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "731cf8ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]: sparse_operation_kit is imported\n"
     ]
    }
   ],
   "source": [
    "import sys, os, json\n",
    "import sparse_operation_kit as sok\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "# import utility python script\n",
    "sys.path.append(\"../unit_test/test_scripts/tf2/\")\n",
    "import utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4cc8918d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]: There are 8 GPUs in total\n"
     ]
    }
   ],
   "source": [
    "total_gpu_num = utils.get_local_gpu_count()\n",
    "print(\"[INFO]: There are %d GPUs in total\" %total_gpu_num)\n",
    "if (total_gpu_num % args[\"worker_num\"] != 0):\n",
    "    raise RuntimeError(\"total_gpu_num:%d is not divisible by workers_num: %d\" %(total_gpu_num, args[\"worker_num\"]))\n",
    "    \n",
    "per_worker_gpu_num = total_gpu_num // args[\"worker_num\"]\n",
    "args[\"local_gpu_num\"] = per_worker_gpu_num # the number of avaiable GPUs in each process"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e20add6",
   "metadata": {},
   "source": [
    "Secondly, define DNN model using Tensorflow and SOK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7258315e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TfDemo(tf.keras.models.Model):\n",
    "    def __init__(self, \n",
    "                 init_tensors, \n",
    "                 combiner, \n",
    "                 global_batch_size,\n",
    "                 slot_num, \n",
    "                 embedding_vec_size,\n",
    "                 **kwargs):\n",
    "        super(TfDemo, self).__init__(**kwargs)\n",
    "        self.combiner = combiner\n",
    "        self.global_batch_size = global_batch_size\n",
    "        self.slot_num = slot_num\n",
    "        self.embedding_vec_size = embedding_vec_size\n",
    "\n",
    "        self.init_tensors = init_tensors\n",
    "        self.params = tf.Variable(initial_value=tf.concat(self.init_tensors, axis=0))\n",
    "\n",
    "        self.dense_layer = tf.keras.layers.Dense(units=1, activation=None,\n",
    "                                                 kernel_initializer=\"ones\",\n",
    "                                                 bias_initializer=\"zeros\")\n",
    "\n",
    "    def call(self, inputs, training=True):\n",
    "        # [batchsize * slot_num, embedding_vec_size]\n",
    "        embedding_vector = tf.nn.embedding_lookup_sparse(params=self.params, sp_ids=inputs,\n",
    "                                                        sp_weights=None, combiner=self.combiner)\n",
    "\n",
    "        # [batchsize, slot_num * embedding_vec_size]\n",
    "        embedding_vector = tf.reshape(embedding_vector, shape=[self.global_batch_size, self.slot_num * self.embedding_vec_size])\n",
    "        logit = self.dense_layer(embedding_vector)\n",
    "        return logit, embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "48baebb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "class SOKDemo(tf.keras.models.Model):\n",
    "    def __init__(self,\n",
    "                 combiner,\n",
    "                 max_vocabulary_size_per_gpu,\n",
    "                 slot_num,\n",
    "                 max_nnz,\n",
    "                 embedding_vec_size, \n",
    "                 **kwargs):\n",
    "        super(SOKDemo, self).__init__(**kwargs)\n",
    "\n",
    "        self.combiner = combiner\n",
    "        self.max_vocabulary_size_per_gpu = max_vocabulary_size_per_gpu\n",
    "        self.slot_num = slot_num\n",
    "        self.max_nnz = max_nnz\n",
    "        self.embedding_vec_size = embedding_vec_size\n",
    "\n",
    "        self.embedding_layer = sok.DistributedEmbedding(combiner=self.combiner,\n",
    "                                                           max_vocabulary_size_per_gpu=self.max_vocabulary_size_per_gpu,\n",
    "                                                           embedding_vec_size=self.embedding_vec_size,\n",
    "                                                           slot_num=self.slot_num,\n",
    "                                                           max_nnz=self.max_nnz)\n",
    "\n",
    "        self.dense_layer = tf.keras.layers.Dense(units=1, activation=None,\n",
    "                                                 kernel_initializer=\"ones\",\n",
    "                                                 bias_initializer=\"zeros\")\n",
    "\n",
    "    def call(self, inputs, training=True):\n",
    "        # [batchsize, slot_num, embedding_vec_size]\n",
    "        embedding_vector = self.embedding_layer(inputs, training=training)\n",
    "        # [batchsize, slot_num * embedding_vec_size]\n",
    "        embedding_vector = tf.reshape(embedding_vector, shape=[-1, self.slot_num * self.embedding_vec_size])\n",
    "        # [batchsize, 1]\n",
    "        logit = self.dense_layer(embedding_vector)\n",
    "        return logit, embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "10b260bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_tf_demo(args, init_tensors, *random_samples):\n",
    "    dataset = utils.tf_dataset(*random_samples, batchsize=args[\"global_batch_size\"], to_sparse_tensor=True, repeat=1)\n",
    "\n",
    "    loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
    "\n",
    "    tf_demo = TfDemo(init_tensors, args[\"combiner\"], args[\"global_batch_size\"], \n",
    "                     args[\"slot_num\"], args[\"embedding_vec_size\"])\n",
    "\n",
    "    optimizer = utils.get_dense_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "\n",
    "    @tf.function\n",
    "    def _train_step(inputs, labels):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logit, embedding_vector = tf_demo(inputs, training=True)\n",
    "            loss = loss_fn(labels, logit)\n",
    "        grads = tape.gradient(loss, tf_demo.trainable_variables)\n",
    "        optimizer.apply_gradients(zip(grads, tf_demo.trainable_variables))\n",
    "        return logit, embedding_vector\n",
    "\n",
    "    tf_results = list()\n",
    "\n",
    "    for i, (sparse_tensors, labels) in enumerate(dataset):\n",
    "        print(\"-\"*30, str(i), \"-\"*30)\n",
    "        logit, embedding_vector = _train_step(sparse_tensors, labels)\n",
    "        print(\"[INFO]: embedding_vector:\\n\", embedding_vector)\n",
    "        tf_results.append(embedding_vector)\n",
    "\n",
    "    return tf_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe04526d",
   "metadata": {},
   "source": [
    "Thirdly, define multi-node training loop for SOK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ed3f416d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_sok_demo(args, task_id, init_tensors, *random_samples):\n",
    "    physical_devices = tf.config.list_physical_devices('GPU')\n",
    "    print(\"[INFO]: physical_devices on task %d:\" %task_id, physical_devices)\n",
    "    \n",
    "    port = 12345\n",
    "    os.environ[\"TF_CONFIG\"] = json.dumps({\n",
    "        'cluster': {\"worker\": [args[\"ips\"][i] + \":\" + str(port + i) for i in range(args[\"worker_num\"])] },\n",
    "        'task': {\"type\": 'worker', \"index\": task_id}\n",
    "    })\n",
    "    strategy = tf.distribute.MultiWorkerMirroredStrategy()\n",
    "    with strategy.scope():\n",
    "        sok.Init(global_batch_size=args[\"global_batch_size\"])\n",
    "\n",
    "        sok_demo = SOKDemo(combiner=args[\"combiner\"], \n",
    "                            max_vocabulary_size_per_gpu=args[\"max_vocabulary_size_per_gpu\"],\n",
    "                            slot_num=args[\"slot_num\"], max_nnz=args[\"max_nnz\"],\n",
    "                            embedding_vec_size=args[\"embedding_vec_size\"])\n",
    "\n",
    "        emb_opt = utils.get_embedding_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "        dense_opt = utils.get_dense_optimizer(args[\"optimizer\"])(learning_rate=0.1)\n",
    "    \n",
    "    sok_saver = sok.Saver()\n",
    "    sok_saver.load_embedding_values(sok_demo.embedding_layer.embedding_variable, init_tensors)\n",
    "\n",
    "    loss_fn = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)\n",
    "    def _replica_loss(labels, logits):\n",
    "        loss = loss_fn(labels, logits)\n",
    "        return tf.nn.compute_average_loss(loss, global_batch_size=args[\"global_batch_size\"])\n",
    "\n",
    "    @tf.function\n",
    "    def _train_step(inputs, labels):\n",
    "        with tf.GradientTape() as tape:\n",
    "            logit, embedding_vector = sok_demo(inputs, training=True)\n",
    "            loss = _replica_loss(labels, logit)\n",
    "        embedding_variables, other_variable = sok.split_embedding_variable_from_others(sok_demo.trainable_variables)\n",
    "        grads, emb_grads = tape.gradient(loss, [other_variable, embedding_variables])\n",
    "        if \"plugin\" not in args[\"optimizer\"]:\n",
    "            with sok.OptimizerScope(embedding_variables):\n",
    "                emb_opt.apply_gradients(zip(emb_grads, embedding_variables),\n",
    "                                        experimental_aggregate_gradients=False)\n",
    "        else:\n",
    "            emb_opt.apply_gradients(zip(emb_grads, embedding_variables),\n",
    "                                    experimental_aggregate_gradients=False)\n",
    "        dense_opt.apply_gradients(zip(grads, other_variable))\n",
    "        return logit, embedding_vector\n",
    "\n",
    "    sok_results = list()\n",
    "\n",
    "    def _dataset_fn(input_context):\n",
    "        replica_batch_size = input_context.get_per_replica_batch_size(args[\"global_batch_size\"])\n",
    "        dataset = utils.tf_dataset(*random_samples, batchsize=replica_batch_size, to_sparse_tensor=True, repeat=1)\n",
    "        # because each worker has its own data source, so that no need to shard the dataset.\n",
    "        return dataset\n",
    "\n",
    "    dataset = strategy.distribute_datasets_from_function(_dataset_fn)\n",
    "\n",
    "    for i, (sparse_tensors, replica_labels) in enumerate(dataset):\n",
    "        print(\"-\" * 30, \"step \", str(i), \"-\" * 30)\n",
    "        logit, embedding_vector = strategy.run(_train_step, args=(sparse_tensors, replica_labels))\n",
    "        print(\"[INFO]: embedding_vector\\n\", embedding_vector)\n",
    "        sok_results.append(embedding_vector)\n",
    "\n",
    "    return sok_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "191288bc",
   "metadata": {},
   "source": [
    "Fourthly, define subprocess work function to simulate multi-node synchronized training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6a4ee6ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compare_sok_with_tf(args, task_id):\n",
    "    if (args[\"global_batch_size\"] % args[\"local_gpu_num\"] != 0):\n",
    "        raise ValueError(\"global_batch_size: %d is not divisible by local_gpu_num: %d\"\n",
    "                            %(args[\"global_batch_size\"], args[\"local_gpu_num\"]))\n",
    "    if (args[\"global_batch_size\"] % args[\"worker_num\"] != 0):\n",
    "        raise ValueError(\"global_batch_size: %d is not divisible by worker_num: %d\"\n",
    "                            %(args[\"global_batch_size\"], args[\"worker_num\"]))\n",
    "\n",
    "    # each worker generate different dataset\n",
    "    worker_batch_size = args[\"global_batch_size\"] // args[\"worker_num\"]\n",
    "    random_samples_local = utils.generate_random_samples(num_of_samples=worker_batch_size * args[\"iter_num\"],\n",
    "                                                         vocabulary_size=args[\"local_gpu_num\"] * args[\"max_vocabulary_size_per_gpu\"] * args[\"worker_num\"],\n",
    "                                                         slot_num=args[\"slot_num\"],\n",
    "                                                         max_nnz=args[\"max_nnz\"])\n",
    "    utils.save_to_file(r\"./random_samples_\" + str(task_id) + r\".file\", *random_samples_local)\n",
    "\n",
    "    # each worker generate same init tensors, because each worker will do the filtering by itself.\n",
    "    init_tensors = utils.get_ones_tensor(max_vocab_size_per_gpu=args[\"max_vocabulary_size_per_gpu\"],\n",
    "                                            embedding_vec_size=args[\"embedding_vec_size\"],\n",
    "                                            num=args[\"local_gpu_num\"] * args[\"worker_num\"])\n",
    "\n",
    "    sok_results_local = test_sok_demo(args, task_id, init_tensors, *random_samples_local)\n",
    "    # save the forward embedding vector from different worker to file\n",
    "    utils.save_to_file(r\"./sok_embedding_vectors_\" + str(task_id) + r\".file\", *sok_results_local)\n",
    "\n",
    "    # aggregate dataset from different worker\n",
    "    dataset_filenames = [r\"./random_samples_\" + str(task_id) + r\".file\"\n",
    "                         for task_id in range(args[\"worker_num\"])]\n",
    "    random_samples_total = [list() for _ in range(args[\"iter_num\"])]\n",
    "    random_labels_total = [list() for _ in range(args[\"iter_num\"])]\n",
    "    local_batch_size = args[\"global_batch_size\"] // args[\"worker_num\"]\n",
    "    for work_id in range(args[\"worker_num\"]):\n",
    "        samples, labels = utils.restore_from_file(dataset_filenames[work_id])\n",
    "        for i in range(args[\"iter_num\"]):\n",
    "            random_samples_total[i].extend(samples[i * local_batch_size : (i + 1) * local_batch_size])\n",
    "            random_labels_total[i].extend(labels[i * local_batch_size : (i + 1) * local_batch_size])\n",
    "    random_samples_total = np.concatenate(random_samples_total, axis=0)\n",
    "    random_labels_total = np.concatenate(random_labels_total, axis=0)\n",
    "\n",
    "    tf_results = test_tf_demo(args, init_tensors, random_samples_total, random_labels_total)\n",
    "\n",
    "    # aggregate forward embedding vector from different worker\n",
    "    sok_results_filenames = [r\"./sok_embedding_vectors_\" + str(task_id) + r\".file\"\n",
    "                             for task_id in range(args[\"worker_num\"])]\n",
    "    sok_results_total = list()\n",
    "    for file_name in sok_results_filenames:\n",
    "        sok_results_local = utils.restore_from_file(file_name)\n",
    "        sok_results_total.append(sok_results_local)\n",
    "\n",
    "    if (len(sok_results_total[0]) != len(tf_results)):\n",
    "        raise ValueError(\"The length of results obtained from sok: %d is not equal to that of tensorflow: %d.\"\n",
    "                        %(len(sok_results_total[0]), len(tf_results)))\n",
    "    if (len(tf_results) != args[\"iter_num\"]):\n",
    "        raise ValueError(\"The length of embedding vectors: %d is not equal to iteration number: %d.\"\n",
    "                         %(len(tf_results), args[\"iter_num\"]))\n",
    "\n",
    "    # for i, sok_vector in enumerate(sok_results_total):\n",
    "    for i in range(args[\"iter_num\"]):\n",
    "        if args[\"local_gpu_num\"] != 1:\n",
    "            sok_vector = tf.concat([tf.concat(sok_results_total[task_id][i].values, axis=0)\n",
    "                                    for task_id in range(args[\"worker_num\"])], axis=0)\n",
    "        else:\n",
    "            sok_vector = tf.concat([sok_results_total[task_id][i]\n",
    "                                    for task_id in range(args[\"worker_num\"])], axis=0)\n",
    "        tf.debugging.assert_near(tf.reshape(sok_vector, \n",
    "                                            shape=[-1, tf.shape(sok_vector)[-1]]),\n",
    "                                 tf_results[i],\n",
    "                                 atol=1e-4,\n",
    "                                 rtol=1e-4)\n",
    "\n",
    "    print((\"\\n[INFO]: With MultiWorkerMirroredStrategy, when %d GPUs are used for each node and %d GPUs in total, \"\n",
    "           \"the embedding vectors obtained from TensorFlow and SOK are consistent for %d iterations\")\n",
    "          %(args[\"local_gpu_num\"], args[\"local_gpu_num\"] * args[\"worker_num\"], args[\"iter_num\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606b0a9b",
   "metadata": {},
   "source": [
    "Fifthly, create sub CPU processes to simulate multi-node synchronized training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "71b27069",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO]: on task: 0, its avaiable GPUs are: 0,1,2,3\n",
      "[INFO]: on task: 1, its avaiable GPUs are: 4,5,6,7\n",
      "[INFO]: begin to generate random samples\n",
      "[INFO]: begin to generate random samples\n",
      "[INFO]: generated random samples\n",
      "[INFO]: generated random samples\n",
      "[INFO]: dumpped items to file ./random_samples_1.file\n",
      "[INFO]: physical_devices on task 1: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]\n",
      "[INFO]: dumpped items to file ./random_samples_0.file\n",
      "[INFO]: physical_devices on task 0: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:1', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:2', device_type='GPU'), PhysicalDevice(name='/physical_device:GPU:3', device_type='GPU')]\n",
      "INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:1/device:CPU:0', '/job:worker/replica:0/task:1/device:GPU:0', '/job:worker/replica:0/task:1/device:GPU:1', '/job:worker/replica:0/task:1/device:GPU:2', '/job:worker/replica:0/task:1/device:GPU:3']\n",
      "INFO:tensorflow:Waiting for the cluster, timeout = inf\n",
      "INFO:tensorflow:Enabled multi-worker collective ops with available devices: ['/job:worker/replica:0/task:0/device:CPU:0', '/job:worker/replica:0/task:0/device:GPU:0', '/job:worker/replica:0/task:0/device:GPU:1', '/job:worker/replica:0/task:0/device:GPU:2', '/job:worker/replica:0/task:0/device:GPU:3']\n",
      "INFO:tensorflow:Waiting for the cluster, timeout = inf\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-12-07 04:50:11.099010: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.099107: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14131 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.101215: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.101268: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14631 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:86:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.103196: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.103250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14631 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:89:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.105134: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.105166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14631 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:8a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.135068: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:1/device:GPU:0 with 14131 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:85:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.136853: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:1/device:GPU:1 with 14631 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:86:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.139148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:1/device:GPU:2 with 14631 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:89:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.140869: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:1/device:GPU:3 with 14631 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:8a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.148301: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12345, 1 -> localhost:12346}\n",
      "2021-12-07 04:50:11.151760: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:427] Started server with target: grpc://localhost:12346\n",
      "2021-12-07 04:50:11.171920: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.172004: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14131 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.174024: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.174062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 14631 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:07:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.176643: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.176682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 14631 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.180031: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.\n",
      "2021-12-07 04:50:11.180069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 14631 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0b:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.210303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:0/device:GPU:0 with 14131 MB memory:  -> device: 0, name: Tesla V100-SXM2-16GB, pci bus id: 0000:06:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.211926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:0/device:GPU:1 with 14631 MB memory:  -> device: 1, name: Tesla V100-SXM2-16GB, pci bus id: 0000:07:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.213627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:0/device:GPU:2 with 14631 MB memory:  -> device: 2, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0a:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.215369: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:worker/replica:0/task:0/device:GPU:3 with 14631 MB memory:  -> device: 3, name: Tesla V100-SXM2-16GB, pci bus id: 0000:0b:00.0, compute capability: 7.0\n",
      "2021-12-07 04:50:11.223604: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:272] Initialize GrpcChannelCache for job worker -> {0 -> localhost:12345, 1 -> localhost:12346}\n",
      "2021-12-07 04:50:11.226864: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:427] Started server with target: grpc://localhost:12345\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Cluster is ready.\n",
      "INFO:tensorflow:Cluster is ready.\n",
      "INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:12345', 'localhost:12346']}, task_type = 'worker', task_id = 1, num_workers = 2, local_devices = ('/job:worker/task:1/device:GPU:0', '/job:worker/task:1/device:GPU:1', '/job:worker/task:1/device:GPU:2', '/job:worker/task:1/device:GPU:3'), communication = CommunicationImplementation.AUTO\n",
      "INFO:tensorflow:MultiWorkerMirroredStrategy with cluster_spec = {'worker': ['localhost:12345', 'localhost:12346']}, task_type = 'worker', task_id = 0, num_workers = 2, local_devices = ('/job:worker/task:0/device:GPU:0', '/job:worker/task:0/device:GPU:1', '/job:worker/task:0/device:GPU:2', '/job:worker/task:0/device:GPU:3'), communication = CommunicationImplementation.AUTO\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2021-12-07 04:50:12.670152: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "2021-12-07 04:50:12.713944: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:907] Skipping loop optimization for Merge node with control input: cond/branch_executed/_6\n",
      "2021-12-07 04:50:13.649949: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "2021-12-07 04:50:13.850567: W tensorflow/core/grappler/optimizers/loop_optimizer.cc:907] Skipping loop optimization for Merge node with control input: replica_3/cond_1/branch_executed/_55\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:99] Mapping from local_replica_id to device_id:\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 0 -> 0\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 1 -> 1\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 2 -> 2\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 3 -> 3\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:77] Global seed is 783670919\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:78] Local GPU Count: 4\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:79] Global GPU Count: 8\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:99] Mapping from local_replica_id to device_id:\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 0 -> 0\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 1 -> 1\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 2 -> 2\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:101] 3 -> 3\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:77] Global seed is 783670919\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:78] Local GPU Count: 4\n",
      "2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:79] Global GPU Count: 8\n",
      "2021-12-07 04:50:13.852613: I 2021-sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] 12-07 04:50:Global Replica Id: 3; Local Replica Id: 3\n",
      "13.852613: I 2021-12-07 04:50:13.852613: I 2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 0; Local Replica Id: 0\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 1; Local Replica Id: 1\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 2; Local Replica Id: 2\n",
      "2021-12-07 04:50:13.852613: I 2021-12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 6; Local Replica Id: 22021\n",
      "Global Replica Id: 5; Local Replica Id: 1\n",
      "2021-12-07 04:50:13.852613: I -12-07 04:50:13.852613: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 7; Local Replica Id: 3\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:119] Global Replica Id: 4; Local Replica Id: 0\n",
      "2021-12-07 04:50:17.852617: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:188] All peer to peer access enabled.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/resources/manager.cc:188] All peer to peer access enabled.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_1/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_1/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_2/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_2/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_3/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:126] Created embedding variable whose name is EmbeddingVariable/replica_3/\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/lookuper/distributed.cc:56] max_vocabulary_size_in_total = 8192\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/lookuper/distributed.cc:56] max_vocabulary_size_in_total = 8192\n",
      "2021-12-07 04:50:18.852618: I 2021-12-sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:0102] 7 04:50:18.Variable: EmbeddingVariable on global_replica_id: 3 start initialization\n",
      "852618: I 2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 0 start initialization\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 2 start initialization\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 1 start initialization\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 3 initialization done.\n",
      "2021-12-07 04:50:18.852618: I 2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 0 initialization done.\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 2 initialization done.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 1 initialization done.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 5 start initialization\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 6 start initialization\n",
      "2021-12-20217- 12-040:507: 18.4:85261850: I :18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:102] Variable: EmbeddingVariable on global_replica_id: 7 start initialization\n",
      "Variable: EmbeddingVariable on global_replica_id: 4 start initialization\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 5 initialization done.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 6 initialization done.\n",
      "2021-12-07 04:50:18.852618: I 2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 7 initialization done.\n",
      "sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:119] Variable: EmbeddingVariable on global_replica_id: 4 initialization done.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/facade.cc:260] SparseOperationKit allocated internal memory.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:213] Loading embedding values to Variable: EmbeddingVariable...\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/facade.cc:260] SparseOperationKit allocated internal memory.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:213] Loading embedding values to Variable: EmbeddingVariable...\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:307] Allocated temporary buffer for loading embedding values.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_param.cc:307] Allocated temporary buffer for loading embedding values.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:299] num_total_keys = 8192, while total_max_vocabulary_size = 8192\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:299] num_total_keys = 8192, while total_max_vocabulary_size = 8192\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:349] Worker 0: Start uploading parameters. Total loop_num = 8\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc_impl/embedding/common/src/dumping_functions.cc:349] Worker 1: Start uploading parameters. Total loop_num = 8\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:223] Loaded embedding values to Variable: EmbeddingVariable.\n",
      "2021-12-07 04:50:18.852618: I sparse_operation_kit/kit_cc/kit_cc_infra/src/parameters/raw_manager.cc:223] Loaded embedding values to Variable: EmbeddingVariable.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------------------------------------  step step   00  ------------------------------------------------------------\n",
      "\n",
      "INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 4, group_size = 8, implementation = AUTO, num_packs = 1\n",
      "INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 4, group_size = 8, implementation = AUTO, num_packs = 1\n",
      "INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 4, group_size = 8, implementation = AUTO, num_packs = 1\n",
      "INFO:tensorflow:Collective all_reduce tensors: 2 all_reduces, num_devices = 4, group_size = 8, implementation = AUTO, num_packs = 1\n",
      "[INFO]: embedding_vector\n",
      " [INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------------------------------------  step step   11  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.9005065  0.9005065  0.9005065  ... 0.9003671  0.9003671  0.9003671 ]\n",
      " [0.90058446 0.90058446 0.90058446 ... 0.9005494  0.9005494  0.9005494 ]\n",
      " [0.90049314 0.90049314 0.90049314 ... 0.9005217  0.9005217  0.9005217 ]\n",
      " ...\n",
      " [0.90052307 0.90052307 0.90052307 ... 0.9004429  0.9004429  0.9004429 ]\n",
      " [0.9005232  0.9005232  0.9005232  ... 0.9004705  0.9004705  0.9004705 ]\n",
      " [0.9005294  0.9005294  0.9005294  ... 0.9005312  0.9005312  0.9005312 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.9005344  0.9005344  0.9005344  ... 0.9005936  0.9005936  0.9005936 ]\n",
      " [0.90065706 0.90065706 0.90065706 ... 0.90054965 0.90054965 0.90054965]\n",
      " [0.9005685  0.9005685  0.9005685  ... 0.9004778  0.9004778  0.9004778 ]\n",
      " ...\n",
      " [0.9005089  0.9005089  0.9005089  ... 0.9004933  0.9004933  0.9004933 ]\n",
      " [0.9005389  0.9005389  0.9005389  ... 0.9004872  0.9004872  0.9004872 ]\n",
      " [0.90050155 0.90050155 0.90050155 ... 0.90050393 0.90050393 0.90050393]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.9005804  0.9005804  0.9005804  ... 0.9004437  0.9004437  0.9004437 ]\n",
      " [0.90049493 0.90049493 0.90049493 ... 0.9005603  0.9005603  0.9005603 ]\n",
      " [0.9004797  0.9004797  0.9004797  ... 0.9005236  0.9005236  0.9005236 ]\n",
      " ...\n",
      " [0.9005908  0.9005908  0.9005908  ... 0.90055656 0.90055656 0.90055656]\n",
      " [0.9005472  0.9005472  0.9005472  ... 0.90048283 0.90048283 0.90048283]\n",
      " [0.9004255  0.9004255  0.9004255  ... 0.9004461  0.9004461  0.9004461 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.9005221  0.9005221  0.9005221  ... 0.90047574 0.90047574 0.90047574]\n",
      " [0.9006134  0.9006134  0.9006134  ... 0.9005297  0.9005297  0.9005297 ]\n",
      " [0.90054244 0.90054244 0.90054244 ... 0.90046185 0.90046185 0.90046185]\n",
      " ...\n",
      " [0.9005021  0.9005021  0.9005021  ... 0.90063363 0.90063363 0.90063363]\n",
      " [0.90059286 0.90059286 0.90059286 ... 0.90048337 0.90048337 0.90048337]\n",
      " [0.90051436 0.90051436 0.90051436 ... 0.90054846 0.90054846 0.90054846]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.9005291  0.9005291  0.9005291  ... 0.9005343  0.9005343  0.9005343 ]\n",
      " [0.90056205 0.90056205 0.90056205 ... 0.90053666 0.90053666 0.90053666]\n",
      " [0.90047276 0.90047276 0.90047276 ... 0.9006467  0.9006467  0.9006467 ]\n",
      " ...\n",
      " [0.9005763  0.9005763  0.9005763  ... 0.90059566 0.90059566 0.90059566]\n",
      " [0.90045136 0.90045136 0.90045136 ... 0.9004456  0.9004456  0.9004456 ]\n",
      " [0.90051764 0.90051764 0.90051764 ... 0.9005182  0.9005182  0.9005182 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.90052104 0.90052104 0.90052104 ... 0.9005375  0.9005375  0.9005375 ]\n",
      " [0.90046096 0.90046096 0.90046096 ... 0.90055823 0.90055823 0.90055823]\n",
      " [0.9005203  0.9005203  0.9005203  ... 0.90048337 0.90048337 0.90048337]\n",
      " ...\n",
      " [0.900526   0.900526   0.900526   ... 0.900494   0.900494   0.900494  ]\n",
      " [0.9005646  0.9005646  0.9005646  ... 0.90043515 0.90043515 0.90043515]\n",
      " [0.9004923  0.9004923  0.9004923  ... 0.9004866  0.9004866  0.9004866 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.90058863 0.90058863 0.90058863 ... 0.9004769  0.9004769  0.9004769 ]\n",
      " [0.9004606  0.9004606  0.9004606  ... 0.9005701  0.9005701  0.9005701 ]\n",
      " [0.900551   0.900551   0.900551   ... 0.90046227 0.90046227 0.90046227]\n",
      " ...\n",
      " [0.90052    0.90052    0.90052    ... 0.90048164 0.90048164 0.90048164]\n",
      " [0.9006288  0.9006288  0.9006288  ... 0.9005226  0.9005226  0.9005226 ]\n",
      " [0.90046704 0.90046704 0.90046704 ... 0.9004878  0.9004878  0.9004878 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.90052694 0.90052694 0.90052694 ... 0.90054953 0.90054953 0.90054953]\n",
      " [0.90044236 0.90044236 0.90044236 ... 0.9005679  0.9005679  0.9005679 ]\n",
      " [0.90059835 0.90059835 0.90059835 ... 0.9004567  0.9004567  0.9004567 ]\n",
      " ...\n",
      " [0.90051305 0.90051305 0.90051305 ... 0.900492   0.900492   0.900492  ]\n",
      " [0.90054125 0.90054125 0.90054125 ... 0.9005698  0.9005698  0.9005698 ]\n",
      " [0.9005252  0.9005252  0.9005252  ... 0.900566   0.900566   0.900566  ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   22  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.80091035 0.80091035 0.80091035 ... 0.80187684 0.80187684 0.80187684]\n",
      " [0.8018269  0.8018269  0.8018269  ... 0.80089664 0.80089664 0.80089664]\n",
      " [0.80132854 0.80132854 0.80132854 ... 0.80213344 0.80213344 0.80213344]\n",
      " ...\n",
      " [0.8012363  0.8012363  0.8012363  ... 0.80205655 0.80205655 0.80205655]\n",
      " [0.8020546  0.8020546  0.8020546  ... 0.80527157 0.80527157 0.80527157]\n",
      " [0.8014924  0.8014924  0.8014924  ... 0.80085516 0.80085516 0.80085516]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.8024522  0.8024522  0.8024522  ... 0.80182743 0.80182743 0.80182743]\n",
      " [0.8013419  0.8013419  0.8013419  ... 0.80185556 0.80185556 0.80185556]\n",
      " [0.8028189  0.8028189  0.8028189  ... 0.80114806 0.80114806 0.80114806]\n",
      " ...\n",
      " [0.8010341  0.8010341  0.8010341  ... 0.8014995  0.8014995  0.8014995 ]\n",
      " [0.8025521  0.8025521  0.8025521  ... 0.8016705  0.8016705  0.8016705 ]\n",
      " [0.80115086 0.80115086 0.80115086 ... 0.80315006 0.80315006 0.80315006]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.8016602  0.8016602  0.8016602  ... 0.8011329  0.8011329  0.8011329 ]\n",
      " [0.80096453 0.80096453 0.80096453 ... 0.80118626 0.80118626 0.80118626]\n",
      " [0.80126405 0.80126405 0.80126405 ... 0.80133843 0.80133843 0.80133843]\n",
      " ...\n",
      " [0.8008437  0.8008437  0.8008437  ... 0.8008878  0.8008878  0.8008878 ]\n",
      " [0.80136    0.80136    0.80136    ... 0.8034353  0.8034353  0.8034353 ]\n",
      " [0.80130184 0.80130184 0.80130184 ... 0.8021264  0.8021264  0.8021264 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.8019638  0.8019638  0.8019638  ... 0.80125237 0.80125237 0.80125237]\n",
      " [0.80082095 0.80082095 0.80082095 ... 0.80186415 0.80186415 0.80186415]\n",
      " [0.80166626 0.80166626 0.80166626 ... 0.8012723  0.8012723  0.8012723 ]\n",
      " ...\n",
      " [0.802148   0.802148   0.802148   ... 0.80141973 0.80141973 0.80141973]\n",
      " [0.80086744 0.80086744 0.80086744 ... 0.8021209  0.8021209  0.8021209 ]\n",
      " [0.8010088  0.8010088  0.8010088  ... 0.8011025  0.8011025  0.8011025 ]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.8010758  0.8010758  0.8010758  ... 0.80136186 0.80136186 0.80136186]\n",
      " [0.8019467  0.8019467  0.8019467  ... 0.8021935  0.8021935  0.8021935 ]\n",
      " [0.8017717  0.8017717  0.8017717  ... 0.80201644 0.80201644 0.80201644]\n",
      " ...\n",
      " [0.8017008  0.8017008  0.8017008  ... 0.80107605 0.80107605 0.80107605]\n",
      " [0.8011438  0.8011438  0.8011438  ... 0.8013276  0.8013276  0.8013276 ]\n",
      " [0.8024448  0.8024448  0.8024448  ... 0.8031342  0.8031342  0.8031342 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.8012887  0.8012887  0.8012887  ... 0.80096495 0.80096495 0.80096495]\n",
      " [0.8042176  0.8042176  0.8042176  ... 0.80138516 0.80138516 0.80138516]\n",
      " [0.8019333  0.8019333  0.8019333  ... 0.80153835 0.80153835 0.80153835]\n",
      " ...\n",
      " [0.8017067  0.8017067  0.8017067  ... 0.80109566 0.80109566 0.80109566]\n",
      " [0.8018422  0.8018422  0.8018422  ... 0.80205244 0.80205244 0.80205244]\n",
      " [0.80099267 0.80099267 0.80099267 ... 0.8023248  0.8023248  0.8023248 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.8006941  0.8006941  0.8006941  ... 0.8023224  0.8023224  0.8023224 ]\n",
      " [0.8013195  0.8013195  0.8013195  ... 0.8010096  0.8010096  0.8010096 ]\n",
      " [0.8016299  0.8016299  0.8016299  ... 0.80235636 0.80235636 0.80235636]\n",
      " ...\n",
      " [0.80148035 0.80148035 0.80148035 ... 0.80199945 0.80199945 0.80199945]\n",
      " [0.8013142  0.8013142  0.8013142  ... 0.8008978  0.8008978  0.8008978 ]\n",
      " [0.8010081  0.8010081  0.8010081  ... 0.8011515  0.8011515  0.8011515 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.80157584 0.80157584 0.80157584 ... 0.8010095  0.8010095  0.8010095 ]\n",
      " [0.80279016 0.80279016 0.80279016 ... 0.803381   0.803381   0.803381  ]\n",
      " [0.80357224 0.80357224 0.80357224 ... 0.8008934  0.8008934  0.8008934 ]\n",
      " ...\n",
      " [0.80097556 0.80097556 0.80097556 ... 0.80161464 0.80161464 0.80161464]\n",
      " [0.80109143 0.80109143 0.80109143 ... 0.8018794  0.8018794  0.8018794 ]\n",
      " [0.80112785 0.80112785 0.80112785 ... 0.8019363  0.8019363  0.8019363 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   33  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.70541656 0.70541656 0.70541656 ... 0.7036747  0.7036747  0.7036747 ]\n",
      " [0.70298237 0.70298237 0.70298237 ... 0.703443   0.703443   0.703443  ]\n",
      " [0.7038187  0.7038187  0.7038187  ... 0.70313454 0.70313454 0.70313454]\n",
      " ...\n",
      " [0.7020575  0.7020575  0.7020575  ... 0.7012721  0.7012721  0.7012721 ]\n",
      " [0.7031751  0.7031751  0.7031751  ... 0.7017982  0.7017982  0.7017982 ]\n",
      " [0.70441145 0.70441145 0.70441145 ... 0.70355314 0.70355314 0.70355314]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.70513386 0.70513386 0.70513386 ... 0.7018467  0.7018467  0.7018467 ]\n",
      " [0.7037107  0.7037107  0.7037107  ... 0.7041372  0.7041372  0.7041372 ]\n",
      " [0.7023498  0.7023498  0.7023498  ... 0.704746   0.704746   0.704746  ]\n",
      " ...\n",
      " [0.70263267 0.70263267 0.70263267 ... 0.70360756 0.70360756 0.70360756]\n",
      " [0.70339537 0.70339537 0.70339537 ... 0.703791   0.703791   0.703791  ]\n",
      " [0.70285505 0.70285505 0.70285505 ... 0.7035917  0.7035917  0.7035917 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.70294404 0.70294404 0.70294404 ... 0.70177853 0.70177853 0.70177853]\n",
      " [0.7024561  0.7024561  0.7024561  ... 0.703889   0.703889   0.703889  ]\n",
      " [0.7028274  0.7028274  0.7028274  ... 0.7010149  0.7010149  0.7010149 ]\n",
      " ...\n",
      " [0.7032586  0.7032586  0.7032586  ... 0.70336825 0.70336825 0.70336825]\n",
      " [0.70156896 0.70156896 0.70156896 ... 0.70484173 0.70484173 0.70484173]\n",
      " [0.7033651  0.7033651  0.7033651  ... 0.7035479  0.7035479  0.7035479 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.70291054 0.70291054 0.70291054 ... 0.70419884 0.70419884 0.70419884]\n",
      " [0.70590985 0.70590985 0.70590985 ... 0.70558226 0.70558226 0.70558226]\n",
      " [0.7044353  0.7044353  0.7044353  ... 0.70099133 0.70099133 0.70099133]\n",
      " ...\n",
      " [0.706217   0.706217   0.706217   ... 0.7021504  0.7021504  0.7021504 ]\n",
      " [0.70343363 0.70343363 0.70343363 ... 0.7082239  0.7082239  0.7082239 ]\n",
      " [0.70296824 0.70296824 0.70296824 ... 0.70155525 0.70155525 0.70155525]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.70298064 0.70298064 0.70298064 ... 0.7014903  0.7014903  0.7014903 ]\n",
      " [0.7035034  0.7035034  0.7035034  ... 0.7024431  0.7024431  0.7024431 ]\n",
      " [0.7033452  0.7033452  0.7033452  ... 0.7089113  0.7089113  0.7089113 ]\n",
      " ...\n",
      " [0.70500076 0.70500076 0.70500076 ... 0.70410526 0.70410526 0.70410526]\n",
      " [0.7032043  0.7032043  0.7032043  ... 0.7064058  0.7064058  0.7064058 ]\n",
      " [0.7036304  0.7036304  0.7036304  ... 0.7030015  0.7030015  0.7030015 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.7026798  0.7026798  0.7026798  ... 0.7042356  0.7042356  0.7042356 ]\n",
      " [0.70326644 0.70326644 0.70326644 ... 0.70467246 0.70467246 0.70467246]\n",
      " [0.705318   0.705318   0.705318   ... 0.70233506 0.70233506 0.70233506]\n",
      " ...\n",
      " [0.70432425 0.70432425 0.70432425 ... 0.7036905  0.7036905  0.7036905 ]\n",
      " [0.701897   0.701897   0.701897   ... 0.7037842  0.7037842  0.7037842 ]\n",
      " [0.7036492  0.7036492  0.7036492  ... 0.7057413  0.7057413  0.7057413 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.70624083 0.70624083 0.70624083 ... 0.7028823  0.7028823  0.7028823 ]\n",
      " [0.70217466 0.70217466 0.70217466 ... 0.7045318  0.7045318  0.7045318 ]\n",
      " [0.7028299  0.7028299  0.7028299  ... 0.7020103  0.7020103  0.7020103 ]\n",
      " ...\n",
      " [0.7039519  0.7039519  0.7039519  ... 0.7020386  0.7020386  0.7020386 ]\n",
      " [0.7028528  0.7028528  0.7028528  ... 0.7047748  0.7047748  0.7047748 ]\n",
      " [0.7085869  0.7085869  0.7085869  ... 0.7036462  0.7036462  0.7036462 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.70343816 0.70343816 0.70343816 ... 0.7028142  0.7028142  0.7028142 ]\n",
      " [0.7014174  0.7014174  0.7014174  ... 0.7056713  0.7056713  0.7056713 ]\n",
      " [0.7031522  0.7031522  0.7031522  ... 0.70561415 0.70561415 0.70561415]\n",
      " ...\n",
      " [0.702861   0.702861   0.702861   ... 0.70316935 0.70316935 0.70316935]\n",
      " [0.70522946 0.70522946 0.70522946 ... 0.70421135 0.70421135 0.70421135]\n",
      " [0.7024635  0.7024635  0.7024635  ... 0.7026086  0.7026086  0.7026086 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   44  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.60456544 0.60456544 0.60456544 ... 0.6037487  0.6037487  0.6037487 ]\n",
      " [0.60749495 0.60749495 0.60749495 ... 0.6064673  0.6064673  0.6064673 ]\n",
      " [0.60752    0.60752    0.60752    ... 0.6063677  0.6063677  0.6063677 ]\n",
      " ...\n",
      " [0.60513824 0.60513824 0.60513824 ... 0.6088296  0.6088296  0.6088296 ]\n",
      " [0.6084243  0.6084243  0.6084243  ... 0.60536164 0.60536164 0.60536164]\n",
      " [0.60568666 0.60568666 0.60568666 ... 0.6064166  0.6064166  0.6064166 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.6069003  0.6069003  0.6069003  ... 0.6063498  0.6063498  0.6063498 ]\n",
      " [0.6197984  0.6197984  0.6197984  ... 0.6039369  0.6039369  0.6039369 ]\n",
      " [0.6057131  0.6057131  0.6057131  ... 0.60780036 0.60780036 0.60780036]\n",
      " ...\n",
      " [0.60548294 0.60548294 0.60548294 ... 0.60700583 0.60700583 0.60700583]\n",
      " [0.60756725 0.60756725 0.60756725 ... 0.60638607 0.60638607 0.60638607]\n",
      " [0.60661227 0.60661227 0.60661227 ... 0.60659397 0.60659397 0.60659397]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.60972035 0.60972035 0.60972035 ... 0.61012965 0.61012965 0.61012965]\n",
      " [0.60629916 0.60629916 0.60629916 ... 0.6062427  0.6062427  0.6062427 ]\n",
      " [0.6033044  0.6033044  0.6033044  ... 0.60401356 0.60401356 0.60401356]\n",
      " ...\n",
      " [0.61241317 0.61241317 0.61241317 ... 0.6064001  0.6064001  0.6064001 ]\n",
      " [0.6029197  0.6029197  0.6029197  ... 0.6086259  0.6086259  0.6086259 ]\n",
      " [0.6060879  0.6060879  0.6060879  ... 0.607451   0.607451   0.607451  ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.61120987 0.61120987 0.61120987 ... 0.6072715  0.6072715  0.6072715 ]\n",
      " [0.60435146 0.60435146 0.60435146 ... 0.6035353  0.6035353  0.6035353 ]\n",
      " [0.6047001  0.6047001  0.6047001  ... 0.6045737  0.6045737  0.6045737 ]\n",
      " ...\n",
      " [0.6061349  0.6061349  0.6061349  ... 0.6064899  0.6064899  0.6064899 ]\n",
      " [0.60596484 0.60596484 0.60596484 ... 0.60521007 0.60521007 0.60521007]\n",
      " [0.60520244 0.60520244 0.60520244 ... 0.6082902  0.6082902  0.6082902 ]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.60496116 0.60496116 0.60496116 ... 0.60659146 0.60659146 0.60659146]\n",
      " [0.6058155  0.6058155  0.6058155  ... 0.60560036 0.60560036 0.60560036]\n",
      " [0.606745   0.606745   0.606745   ... 0.60665023 0.60665023 0.60665023]\n",
      " ...\n",
      " [0.6053973  0.6053973  0.6053973  ... 0.6064053  0.6064053  0.6064053 ]\n",
      " [0.60366404 0.60366404 0.60366404 ... 0.60544264 0.60544264 0.60544264]\n",
      " [0.6058608  0.6058608  0.6058608  ... 0.6104212  0.6104212  0.6104212 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.6063523  0.6063523  0.6063523  ... 0.60864097 0.60864097 0.60864097]\n",
      " [0.6086781  0.6086781  0.6086781  ... 0.60800236 0.60800236 0.60800236]\n",
      " [0.60317314 0.60317314 0.60317314 ... 0.6095214  0.6095214  0.6095214 ]\n",
      " ...\n",
      " [0.6087903  0.6087903  0.6087903  ... 0.6043147  0.6043147  0.6043147 ]\n",
      " [0.60209495 0.60209495 0.60209495 ... 0.6071436  0.6071436  0.6071436 ]\n",
      " [0.6031012  0.6031012  0.6031012  ... 0.6070004  0.6070004  0.6070004 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.6051136  0.6051136  0.6051136  ... 0.60493886 0.60493886 0.60493886]\n",
      " [0.6074388  0.6074388  0.6074388  ... 0.60758054 0.60758054 0.60758054]\n",
      " [0.60571265 0.60571265 0.60571265 ... 0.6088873  0.6088873  0.6088873 ]\n",
      " ...\n",
      " [0.6113419  0.6113419  0.6113419  ... 0.60720146 0.60720146 0.60720146]\n",
      " [0.60433614 0.60433614 0.60433614 ... 0.604125   0.604125   0.604125  ]\n",
      " [0.6066602  0.6066602  0.6066602  ... 0.60827315 0.60827315 0.60827315]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.6072669  0.6072669  0.6072669  ... 0.60766083 0.60766083 0.60766083]\n",
      " [0.6038309  0.6038309  0.6038309  ... 0.60599875 0.60599875 0.60599875]\n",
      " [0.6041541  0.6041541  0.6041541  ... 0.6058229  0.6058229  0.6058229 ]\n",
      " ...\n",
      " [0.605454   0.605454   0.605454   ... 0.60733056 0.60733056 0.60733056]\n",
      " [0.605268   0.605268   0.605268   ... 0.60525435 0.60525435 0.60525435]\n",
      " [0.607793   0.607793   0.607793   ... 0.6090448  0.6090448  0.6090448 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   55  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.512373   0.512373   0.512373   ... 0.50540996 0.50540996 0.50540996]\n",
      " [0.5114419  0.5114419  0.5114419  ... 0.50784224 0.50784224 0.50784224]\n",
      " [0.5094663  0.5094663  0.5094663  ... 0.5131079  0.5131079  0.5131079 ]\n",
      " ...\n",
      " [0.51152116 0.51152116 0.51152116 ... 0.5191655  0.5191655  0.5191655 ]\n",
      " [0.513106   0.513106   0.513106   ... 0.51256424 0.51256424 0.51256424]\n",
      " [0.5165243  0.5165243  0.5165243  ... 0.5093701  0.5093701  0.5093701 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.51201713 0.51201713 0.51201713 ... 0.5136352  0.5136352  0.5136352 ]\n",
      " [0.5089708  0.5089708  0.5089708  ... 0.52010214 0.52010214 0.52010214]\n",
      " [0.50950354 0.50950354 0.50950354 ... 0.5139234  0.5139234  0.5139234 ]\n",
      " ...\n",
      " [0.5103274  0.5103274  0.5103274  ... 0.5121634  0.5121634  0.5121634 ]\n",
      " [0.5121676  0.5121676  0.5121676  ... 0.5119947  0.5119947  0.5119947 ]\n",
      " [0.5121393  0.5121393  0.5121393  ... 0.5138248  0.5138248  0.5138248 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.5107837  0.5107837  0.5107837  ... 0.50861514 0.50861514 0.50861514]\n",
      " [0.50833863 0.50833863 0.50833863 ... 0.51145506 0.51145506 0.51145506]\n",
      " [0.5096166  0.5096166  0.5096166  ... 0.5111596  0.5111596  0.5111596 ]\n",
      " ...\n",
      " [0.5126155  0.5126155  0.5126155  ... 0.5098642  0.5098642  0.5098642 ]\n",
      " [0.51348567 0.51348567 0.51348567 ... 0.5113246  0.5113246  0.5113246 ]\n",
      " [0.5129145  0.5129145  0.5129145  ... 0.5084769  0.5084769  0.5084769 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.5080177  0.5080177  0.5080177  ... 0.5084405  0.5084405  0.5084405 ]\n",
      " [0.5117988  0.5117988  0.5117988  ... 0.51558703 0.51558703 0.51558703]\n",
      " [0.5131069  0.5131069  0.5131069  ... 0.51138073 0.51138073 0.51138073]\n",
      " ...\n",
      " [0.5048766  0.5048766  0.5048766  ... 0.5125449  0.5125449  0.5125449 ]\n",
      " [0.5108046  0.5108046  0.5108046  ... 0.5068901  0.5068901  0.5068901 ]\n",
      " [0.5158355  0.5158355  0.5158355  ... 0.5170237  0.5170237  0.5170237 ]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.5128385  0.5128385  0.5128385  ... 0.51144886 0.51144886 0.51144886]\n",
      " [0.507567   0.507567   0.507567   ... 0.5113499  0.5113499  0.5113499 ]\n",
      " [0.5051104  0.5051104  0.5051104  ... 0.5122803  0.5122803  0.5122803 ]\n",
      " ...\n",
      " [0.5137641  0.5137641  0.5137641  ... 0.5079634  0.5079634  0.5079634 ]\n",
      " [0.5111262  0.5111262  0.5111262  ... 0.50875914 0.50875914 0.50875914]\n",
      " [0.5178307  0.5178307  0.5178307  ... 0.5147817  0.5147817  0.5147817 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.5082871  0.5082871  0.5082871  ... 0.5140504  0.5140504  0.5140504 ]\n",
      " [0.51116216 0.51116216 0.51116216 ... 0.5132304  0.5132304  0.5132304 ]\n",
      " [0.51287615 0.51287615 0.51287615 ... 0.5163946  0.5163946  0.5163946 ]\n",
      " ...\n",
      " [0.51039183 0.51039183 0.51039183 ... 0.5139927  0.5139927  0.5139927 ]\n",
      " [0.5116324  0.5116324  0.5116324  ... 0.51418114 0.51418114 0.51418114]\n",
      " [0.5165117  0.5165117  0.5165117  ... 0.5069572  0.5069572  0.5069572 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.5142305  0.5142305  0.5142305  ... 0.51688015 0.51688015 0.51688015]\n",
      " [0.5171165  0.5171165  0.5171165  ... 0.5111654  0.5111654  0.5111654 ]\n",
      " [0.5069978  0.5069978  0.5069978  ... 0.5188557  0.5188557  0.5188557 ]\n",
      " ...\n",
      " [0.5094886  0.5094886  0.5094886  ... 0.5125552  0.5125552  0.5125552 ]\n",
      " [0.50921845 0.50921845 0.50921845 ... 0.5106788  0.5106788  0.5106788 ]\n",
      " [0.5056216  0.5056216  0.5056216  ... 0.5190212  0.5190212  0.5190212 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.51092684 0.51092684 0.51092684 ... 0.5082636  0.5082636  0.5082636 ]\n",
      " [0.50967747 0.50967747 0.50967747 ... 0.5084994  0.5084994  0.5084994 ]\n",
      " [0.50794935 0.50794935 0.50794935 ... 0.510271   0.510271   0.510271  ]\n",
      " ...\n",
      " [0.5070789  0.5070789  0.5070789  ... 0.50826585 0.50826585 0.50826585]\n",
      " [0.50743496 0.50743496 0.50743496 ... 0.510599   0.510599   0.510599  ]\n",
      " [0.51036644 0.51036644 0.51036644 ... 0.5116213  0.5116213  0.5116213 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   66  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.41768348 0.41768348 0.41768348 ... 0.41975674 0.41975674 0.41975674]\n",
      " [0.4259813  0.4259813  0.4259813  ... 0.41802302 0.41802302 0.41802302]\n",
      " [0.41776216 0.41776216 0.41776216 ... 0.41541773 0.41541773 0.41541773]\n",
      " ...\n",
      " [0.41558585 0.41558585 0.41558585 ... 0.41533744 0.41533744 0.41533744]\n",
      " [0.4214899  0.4214899  0.4214899  ... 0.41687626 0.41687626 0.41687626]\n",
      " [0.41689354 0.41689354 0.41689354 ... 0.42061776 0.42061776 0.42061776]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.41867954 0.41867954 0.41867954 ... 0.42062306 0.42062306 0.42062306]\n",
      " [0.41879863 0.41879863 0.41879863 ... 0.40803626 0.40803626 0.40803626]\n",
      " [0.41019744 0.41019744 0.41019744 ... 0.41628775 0.41628775 0.41628775]\n",
      " ...\n",
      " [0.41876554 0.41876554 0.41876554 ... 0.4118735  0.4118735  0.4118735 ]\n",
      " [0.41848746 0.41848746 0.41848746 ... 0.42295948 0.42295948 0.42295948]\n",
      " [0.42362142 0.42362142 0.42362142 ... 0.416808   0.416808   0.416808  ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.41520452 0.41520452 0.41520452 ... 0.42005712 0.42005712 0.42005712]\n",
      " [0.41711548 0.41711548 0.41711548 ... 0.42371792 0.42371792 0.42371792]\n",
      " [0.4201069  0.4201069  0.4201069  ... 0.4172784  0.4172784  0.4172784 ]\n",
      " ...\n",
      " [0.41733855 0.41733855 0.41733855 ... 0.41864216 0.41864216 0.41864216]\n",
      " [0.42242104 0.42242104 0.42242104 ... 0.42542762 0.42542762 0.42542762]\n",
      " [0.42285103 0.42285103 0.42285103 ... 0.41326225 0.41326225 0.41326225]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.4120284  0.4120284  0.4120284  ... 0.41650197 0.41650197 0.41650197]\n",
      " [0.4169441  0.4169441  0.4169441  ... 0.42075357 0.42075357 0.42075357]\n",
      " [0.41944402 0.41944402 0.41944402 ... 0.4153265  0.4153265  0.4153265 ]\n",
      " ...\n",
      " [0.4135158  0.4135158  0.4135158  ... 0.4176055  0.4176055  0.4176055 ]\n",
      " [0.4147852  0.4147852  0.4147852  ... 0.41469485 0.41469485 0.41469485]\n",
      " [0.41690385 0.41690385 0.41690385 ... 0.41736817 0.41736817 0.41736817]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.41855583 0.41855583 0.41855583 ... 0.4185282  0.4185282  0.4185282 ]\n",
      " [0.41285908 0.41285908 0.41285908 ... 0.41414613 0.41414613 0.41414613]\n",
      " [0.42163068 0.42163068 0.42163068 ... 0.41743124 0.41743124 0.41743124]\n",
      " ...\n",
      " [0.417592   0.417592   0.417592   ... 0.41985178 0.41985178 0.41985178]\n",
      " [0.41547334 0.41547334 0.41547334 ... 0.4225157  0.4225157  0.4225157 ]\n",
      " [0.41381347 0.41381347 0.41381347 ... 0.42141372 0.42141372 0.42141372]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.4239341  0.4239341  0.4239341  ... 0.4192538  0.4192538  0.4192538 ]\n",
      " [0.4242925  0.4242925  0.4242925  ... 0.41270453 0.41270453 0.41270453]\n",
      " [0.41973093 0.41973093 0.41973093 ... 0.4197301  0.4197301  0.4197301 ]\n",
      " ...\n",
      " [0.4161967  0.4161967  0.4161967  ... 0.4190143  0.4190143  0.4190143 ]\n",
      " [0.4124771  0.4124771  0.4124771  ... 0.4176667  0.4176667  0.4176667 ]\n",
      " [0.41803324 0.41803324 0.41803324 ... 0.41557956 0.41557956 0.41557956]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.41999972 0.41999972 0.41999972 ... 0.4201511  0.4201511  0.4201511 ]\n",
      " [0.4136402  0.4136402  0.4136402  ... 0.41607118 0.41607118 0.41607118]\n",
      " [0.4187832  0.4187832  0.4187832  ... 0.41651496 0.41651496 0.41651496]\n",
      " ...\n",
      " [0.4190906  0.4190906  0.4190906  ... 0.41791138 0.41791138 0.41791138]\n",
      " [0.4102192  0.4102192  0.4102192  ... 0.41782904 0.41782904 0.41782904]\n",
      " [0.4185912  0.4185912  0.4185912  ... 0.41961563 0.41961563 0.41961563]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.41511542 0.41511542 0.41511542 ... 0.41626757 0.41626757 0.41626757]\n",
      " [0.41653678 0.41653678 0.41653678 ... 0.423917   0.423917   0.423917  ]\n",
      " [0.4187376  0.4187376  0.4187376  ... 0.41893417 0.41893417 0.41893417]\n",
      " ...\n",
      " [0.41167247 0.41167247 0.41167247 ... 0.41095316 0.41095316 0.41095316]\n",
      " [0.42643577 0.42643577 0.42643577 ... 0.41433492 0.41433492 0.41433492]\n",
      " [0.42021042 0.42021042 0.42021042 ... 0.42626333 0.42626333 0.42626333]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   77  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.33283263 0.33283263 0.33283263 ... 0.3254104  0.3254104  0.3254104 ]\n",
      " [0.32445025 0.32445025 0.32445025 ... 0.3259919  0.3259919  0.3259919 ]\n",
      " [0.33063728 0.33063728 0.33063728 ... 0.3285883  0.3285883  0.3285883 ]\n",
      " ...\n",
      " [0.3153787  0.3153787  0.3153787  ... 0.3302582  0.3302582  0.3302582 ]\n",
      " [0.3269525  0.3269525  0.3269525  ... 0.32785493 0.32785493 0.32785493]\n",
      " [0.32547283 0.32547283 0.32547283 ... 0.32900184 0.32900184 0.32900184]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.33501142 0.33501142 0.33501142 ... 0.33418608 0.33418608 0.33418608]\n",
      " [0.33627507 0.33627507 0.33627507 ... 0.32427096 0.32427096 0.32427096]\n",
      " [0.32041723 0.32041723 0.32041723 ... 0.32473022 0.32473022 0.32473022]\n",
      " ...\n",
      " [0.32242256 0.32242256 0.32242256 ... 0.33118537 0.33118537 0.33118537]\n",
      " [0.3282548  0.3282548  0.3282548  ... 0.33121413 0.33121413 0.33121413]\n",
      " [0.32002625 0.32002625 0.32002625 ... 0.3224963  0.3224963  0.3224963 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.32972342 0.32972342 0.32972342 ... 0.3281535  0.3281535  0.3281535 ]\n",
      " [0.32941753 0.32941753 0.32941753 ... 0.32148278 0.32148278 0.32148278]\n",
      " [0.3255149  0.3255149  0.3255149  ... 0.34170693 0.34170693 0.34170693]\n",
      " ...\n",
      " [0.3295573  0.3295573  0.3295573  ... 0.32374442 0.32374442 0.32374442]\n",
      " [0.32611772 0.32611772 0.32611772 ... 0.33256918 0.33256918 0.33256918]\n",
      " [0.32795614 0.32795614 0.32795614 ... 0.3371187  0.3371187  0.3371187 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.33523726 0.33523726 0.33523726 ... 0.33796966 0.33796966 0.33796966]\n",
      " [0.3284771  0.3284771  0.3284771  ... 0.32863772 0.32863772 0.32863772]\n",
      " [0.33964694 0.33964694 0.33964694 ... 0.3169607  0.3169607  0.3169607 ]\n",
      " ...\n",
      " [0.3229586  0.3229586  0.3229586  ... 0.3325879  0.3325879  0.3325879 ]\n",
      " [0.32801992 0.32801992 0.32801992 ... 0.32656735 0.32656735 0.32656735]\n",
      " [0.3321599  0.3321599  0.3321599  ... 0.31782317 0.31782317 0.31782317]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.3218866  0.3218866  0.3218866  ... 0.3332579  0.3332579  0.3332579 ]\n",
      " [0.32970056 0.32970056 0.32970056 ... 0.32145226 0.32145226 0.32145226]\n",
      " [0.32327256 0.32327256 0.32327256 ... 0.33350977 0.33350977 0.33350977]\n",
      " ...\n",
      " [0.33197206 0.33197206 0.33197206 ... 0.326662   0.326662   0.326662  ]\n",
      " [0.32814062 0.32814062 0.32814062 ... 0.3233876  0.3233876  0.3233876 ]\n",
      " [0.33348405 0.33348405 0.33348405 ... 0.31963855 0.31963855 0.31963855]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.3302578  0.3302578  0.3302578  ... 0.32992625 0.32992625 0.32992625]\n",
      " [0.32305187 0.32305187 0.32305187 ... 0.32731387 0.32731387 0.32731387]\n",
      " [0.32694817 0.32694817 0.32694817 ... 0.3262753  0.3262753  0.3262753 ]\n",
      " ...\n",
      " [0.32920405 0.32920405 0.32920405 ... 0.3314518  0.3314518  0.3314518 ]\n",
      " [0.3265598  0.3265598  0.3265598  ... 0.33012944 0.33012944 0.33012944]\n",
      " [0.33014986 0.33014986 0.33014986 ... 0.3216048  0.3216048  0.3216048 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.33267838 0.33267838 0.33267838 ... 0.3279665  0.3279665  0.3279665 ]\n",
      " [0.32764593 0.32764593 0.32764593 ... 0.3248538  0.3248538  0.3248538 ]\n",
      " [0.326424   0.326424   0.326424   ... 0.317769   0.317769   0.317769  ]\n",
      " ...\n",
      " [0.32280248 0.32280248 0.32280248 ... 0.33023778 0.33023778 0.33023778]\n",
      " [0.32321024 0.32321024 0.32321024 ... 0.33141014 0.33141014 0.33141014]\n",
      " [0.33454096 0.33454096 0.33454096 ... 0.3360346  0.3360346  0.3360346 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.33019644 0.33019644 0.33019644 ... 0.33748603 0.33748603 0.33748603]\n",
      " [0.3245582  0.3245582  0.3245582  ... 0.33086383 0.33086383 0.33086383]\n",
      " [0.3258946  0.3258946  0.3258946  ... 0.33563873 0.33563873 0.33563873]\n",
      " ...\n",
      " [0.33575535 0.33575535 0.33575535 ... 0.32747146 0.32747146 0.32747146]\n",
      " [0.3317665  0.3317665  0.3317665  ... 0.3266688  0.3266688  0.3266688 ]\n",
      " [0.32296035 0.32296035 0.32296035 ... 0.335204   0.335204   0.335204  ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   88  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.24738804 0.24738804 0.24738804 ... 0.24162053 0.24162053 0.24162053]\n",
      " [0.24345806 0.24345806 0.24345806 ... 0.24493822 0.24493822 0.24493822]\n",
      " [0.25446156 0.25446156 0.25446156 ... 0.23449051 0.23449051 0.23449051]\n",
      " ...\n",
      " [0.23982325 0.23982325 0.23982325 ... 0.2363922  0.2363922  0.2363922 ]\n",
      " [0.24601042 0.24601042 0.24601042 ... 0.24584135 0.24584135 0.24584135]\n",
      " [0.24118945 0.24118945 0.24118945 ... 0.24063969 0.24063969 0.24063969]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.24407175 0.24407175 0.24407175 ... 0.24392866 0.24392866 0.24392866]\n",
      " [0.23806566 0.23806566 0.23806566 ... 0.24597108 0.24597108 0.24597108]\n",
      " [0.23490247 0.23490247 0.23490247 ... 0.24233595 0.24233595 0.24233595]\n",
      " ...\n",
      " [0.2351706  0.2351706  0.2351706  ... 0.24046138 0.24046138 0.24046138]\n",
      " [0.23218015 0.23218015 0.23218015 ... 0.2411891  0.2411891  0.2411891 ]\n",
      " [0.23209405 0.23209405 0.23209405 ... 0.2505022  0.2505022  0.2505022 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.24527662 0.24527662 0.24527662 ... 0.25251305 0.25251305 0.25251305]\n",
      " [0.23726802 0.23726802 0.23726802 ... 0.24944861 0.24944861 0.24944861]\n",
      " [0.24375372 0.24375372 0.24375372 ... 0.24298796 0.24298796 0.24298796]\n",
      " ...\n",
      " [0.24218176 0.24218176 0.24218176 ... 0.247892   0.247892   0.247892  ]\n",
      " [0.23067276 0.23067276 0.23067276 ... 0.23380482 0.23380482 0.23380482]\n",
      " [0.23420495 0.23420495 0.23420495 ... 0.23966639 0.23966639 0.23966639]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.23466724 0.23466724 0.23466724 ... 0.24237311 0.24237311 0.24237311]\n",
      " [0.23517627 0.23517627 0.23517627 ... 0.23599386 0.23599386 0.23599386]\n",
      " [0.26888576 0.26888576 0.26888576 ... 0.23991969 0.23991969 0.23991969]\n",
      " ...\n",
      " [0.24935    0.24935    0.24935    ... 0.2331702  0.2331702  0.2331702 ]\n",
      " [0.24274105 0.24274105 0.24274105 ... 0.23342548 0.23342548 0.23342548]\n",
      " [0.25643316 0.25643316 0.25643316 ... 0.23339231 0.23339231 0.23339231]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.23506397 0.23506397 0.23506397 ... 0.23980045 0.23980045 0.23980045]\n",
      " [0.24067622 0.24067622 0.24067622 ... 0.2388439  0.2388439  0.2388439 ]\n",
      " [0.23719361 0.23719361 0.23719361 ... 0.24090225 0.24090225 0.24090225]\n",
      " ...\n",
      " [0.24921389 0.24921389 0.24921389 ... 0.2496515  0.2496515  0.2496515 ]\n",
      " [0.24006933 0.24006933 0.24006933 ... 0.24064118 0.24064118 0.24064118]\n",
      " [0.23475248 0.23475248 0.23475248 ... 0.24043626 0.24043626 0.24043626]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.24576017 0.24576017 0.24576017 ... 0.24465412 0.24465412 0.24465412]\n",
      " [0.24436861 0.24436861 0.24436861 ... 0.24672823 0.24672823 0.24672823]\n",
      " [0.2503479  0.2503479  0.2503479  ... 0.22655067 0.22655067 0.22655067]\n",
      " ...\n",
      " [0.24408603 0.24408603 0.24408603 ... 0.22687855 0.22687855 0.22687855]\n",
      " [0.23251489 0.23251489 0.23251489 ... 0.24404973 0.24404973 0.24404973]\n",
      " [0.26572958 0.26572958 0.26572958 ... 0.2428817  0.2428817  0.2428817 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.24614874 0.24614874 0.24614874 ... 0.25224942 0.25224942 0.25224942]\n",
      " [0.24853182 0.24853182 0.24853182 ... 0.23589443 0.23589443 0.23589443]\n",
      " [0.24297851 0.24297851 0.24297851 ... 0.23133156 0.23133156 0.23133156]\n",
      " ...\n",
      " [0.24098213 0.24098213 0.24098213 ... 0.2472479  0.2472479  0.2472479 ]\n",
      " [0.245562   0.245562   0.245562   ... 0.24883857 0.24883857 0.24883857]\n",
      " [0.2423099  0.2423099  0.2423099  ... 0.23902635 0.23902635 0.23902635]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.25310233 0.25310233 0.25310233 ... 0.2379218  0.2379218  0.2379218 ]\n",
      " [0.25113675 0.25113675 0.25113675 ... 0.23872969 0.23872969 0.23872969]\n",
      " [0.23814502 0.23814502 0.23814502 ... 0.2403785  0.2403785  0.2403785 ]\n",
      " ...\n",
      " [0.2414196  0.2414196  0.2414196  ... 0.24555223 0.24555223 0.24555223]\n",
      " [0.24649382 0.24649382 0.24649382 ... 0.24577668 0.24577668 0.24577668]\n",
      " [0.23521876 0.23521876 0.23521876 ... 0.23737952 0.23737952 0.23737952]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   99  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.15891615 0.15891615 0.15891615 ... 0.16017291 0.16017291 0.16017291]\n",
      " [0.14629072 0.14629072 0.14629072 ... 0.16968808 0.16968808 0.16968808]\n",
      " [0.17505038 0.17505038 0.17505038 ... 0.15785322 0.15785322 0.15785322]\n",
      " ...\n",
      " [0.16383713 0.16383713 0.16383713 ... 0.17393777 0.17393777 0.17393777]\n",
      " [0.1547552  0.1547552  0.1547552  ... 0.16436747 0.16436747 0.16436747]\n",
      " [0.1617353  0.1617353  0.1617353  ... 0.16974723 0.16974723 0.16974723]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.17899156 0.17899156 0.17899156 ... 0.1595509  0.1595509  0.1595509 ]\n",
      " [0.16392446 0.16392446 0.16392446 ... 0.15842228 0.15842228 0.15842228]\n",
      " [0.15969145 0.15969145 0.15969145 ... 0.16888231 0.16888231 0.16888231]\n",
      " ...\n",
      " [0.1650748  0.1650748  0.1650748  ... 0.15895605 0.15895605 0.15895605]\n",
      " [0.15851335 0.15851335 0.15851335 ... 0.16241895 0.16241895 0.16241895]\n",
      " [0.167079   0.167079   0.167079   ... 0.16568473 0.16568473 0.16568473]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.17491628 0.17491628 0.17491628 ... 0.15233807 0.15233807 0.15233807]\n",
      " [0.1606136  0.1606136  0.1606136  ... 0.17713147 0.17713147 0.17713147]\n",
      " [0.16382231 0.16382231 0.16382231 ... 0.16837874 0.16837874 0.16837874]\n",
      " ...\n",
      " [0.17084514 0.17084514 0.17084514 ... 0.16704178 0.16704178 0.16704178]\n",
      " [0.1522342  0.1522342  0.1522342  ... 0.18141201 0.18141201 0.18141201]\n",
      " [0.16247773 0.16247773 0.16247773 ... 0.16011962 0.16011962 0.16011962]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.16655685 0.16655685 0.16655685 ... 0.16682152 0.16682152 0.16682152]\n",
      " [0.15635416 0.15635416 0.15635416 ... 0.15739164 0.15739164 0.15739164]\n",
      " [0.16120115 0.16120115 0.16120115 ... 0.15786412 0.15786412 0.15786412]\n",
      " ...\n",
      " [0.17155384 0.17155384 0.17155384 ... 0.16619815 0.16619815 0.16619815]\n",
      " [0.15813093 0.15813093 0.15813093 ... 0.14438847 0.14438847 0.14438847]\n",
      " [0.15989687 0.15989687 0.15989687 ... 0.15029144 0.15029144 0.15029144]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.16280517 0.16280517 0.16280517 ... 0.17119205 0.17119205 0.17119205]\n",
      " [0.16622345 0.16622345 0.16622345 ... 0.16612375 0.16612375 0.16612375]\n",
      " [0.16616853 0.16616853 0.16616853 ... 0.16513218 0.16513218 0.16513218]\n",
      " ...\n",
      " [0.16025221 0.16025221 0.16025221 ... 0.16074538 0.16074538 0.16074538]\n",
      " [0.1698473  0.1698473  0.1698473  ... 0.15898608 0.15898608 0.15898608]\n",
      " [0.16813955 0.16813955 0.16813955 ... 0.1654239  0.1654239  0.1654239 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.15526274 0.15526274 0.15526274 ... 0.15772258 0.15772258 0.15772258]\n",
      " [0.1680774  0.1680774  0.1680774  ... 0.16474476 0.16474476 0.16474476]\n",
      " [0.1599095  0.1599095  0.1599095  ... 0.15780994 0.15780994 0.15780994]\n",
      " ...\n",
      " [0.16691843 0.16691843 0.16691843 ... 0.15009467 0.15009467 0.15009467]\n",
      " [0.1569644  0.1569644  0.1569644  ... 0.1601957  0.1601957  0.1601957 ]\n",
      " [0.16393164 0.16393164 0.16393164 ... 0.17822221 0.17822221 0.17822221]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.16578574 0.16578574 0.16578574 ... 0.15618083 0.15618083 0.15618083]\n",
      " [0.16512714 0.16512714 0.16512714 ... 0.15743408 0.15743408 0.15743408]\n",
      " [0.15527928 0.15527928 0.15527928 ... 0.17846704 0.17846704 0.17846704]\n",
      " ...\n",
      " [0.1658254  0.1658254  0.1658254  ... 0.16534124 0.16534124 0.16534124]\n",
      " [0.1535913  0.1535913  0.1535913  ... 0.15650633 0.15650633 0.15650633]\n",
      " [0.16531941 0.16531941 0.16531941 ... 0.16055404 0.16055404 0.16055404]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.16313402 0.16313402 0.16313402 ... 0.16572103 0.16572103 0.16572103]\n",
      " [0.15096326 0.15096326 0.15096326 ... 0.16157591 0.16157591 0.16157591]\n",
      " [0.1622975  0.1622975  0.1622975  ... 0.16158578 0.16158578 0.16158578]\n",
      " ...\n",
      " [0.17040727 0.17040727 0.17040727 ... 0.16247071 0.16247071 0.16247071]\n",
      " [0.1658887  0.1658887  0.1658887  ... 0.169308   0.169308   0.169308  ]\n",
      " [0.1542777  0.1542777  0.1542777  ... 0.14368507 0.14368507 0.14368507]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1010  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.07975477 0.07975477 0.07975477 ... 0.09710205 0.09710205 0.09710205]\n",
      " [0.08443781 0.08443781 0.08443781 ... 0.08533795 0.08533795 0.08533795]\n",
      " [0.08612999 0.08612999 0.08612999 ... 0.10071258 0.10071258 0.10071258]\n",
      " ...\n",
      " [0.08962478 0.08962478 0.08962478 ... 0.09560727 0.09560727 0.09560727]\n",
      " [0.08030242 0.08030242 0.08030242 ... 0.09229682 0.09229682 0.09229682]\n",
      " [0.08951589 0.08951589 0.08951589 ... 0.09770642 0.09770642 0.09770642]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.10288849 0.10288849 0.10288849 ... 0.075858   0.075858   0.075858  ]\n",
      " [0.0839217  0.0839217  0.0839217  ... 0.08710175 0.08710175 0.08710175]\n",
      " [0.1041327  0.1041327  0.1041327  ... 0.09110509 0.09110509 0.09110509]\n",
      " ...\n",
      " [0.07411201 0.07411201 0.07411201 ... 0.08562292 0.08562292 0.08562292]\n",
      " [0.10201174 0.10201174 0.10201174 ... 0.08518706 0.08518706 0.08518706]\n",
      " [0.07075051 0.07075051 0.07075051 ... 0.10019736 0.10019736 0.10019736]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.07543776 0.07543776 0.07543776 ... 0.09629688 0.09629688 0.09629688]\n",
      " [0.09882566 0.09882566 0.09882566 ... 0.09373056 0.09373056 0.09373056]\n",
      " [0.08549014 0.08549014 0.08549014 ... 0.10976902 0.10976902 0.10976902]\n",
      " ...\n",
      " [0.08717253 0.08717253 0.08717253 ... 0.0913862  0.0913862  0.0913862 ]\n",
      " [0.09766969 0.09766969 0.09766969 ... 0.10128184 0.10128184 0.10128184]\n",
      " [0.08754413 0.08754413 0.08754413 ... 0.07646793 0.07646793 0.07646793]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.08851591 0.08851591 0.08851591 ... 0.09437674 0.09437674 0.09437674]\n",
      " [0.08826205 0.08826205 0.08826205 ... 0.09801367 0.09801367 0.09801367]\n",
      " [0.09559506 0.09559506 0.09559506 ... 0.08662436 0.08662436 0.08662436]\n",
      " ...\n",
      " [0.09681574 0.09681574 0.09681574 ... 0.0911965  0.0911965  0.0911965 ]\n",
      " [0.09106442 0.09106442 0.09106442 ... 0.07888614 0.07888614 0.07888614]\n",
      " [0.10630825 0.10630825 0.10630825 ... 0.08723697 0.08723697 0.08723697]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.09687094 0.09687094 0.09687094 ... 0.06834569 0.06834569 0.06834569]\n",
      " [0.08395098 0.08395098 0.08395098 ... 0.08104706 0.08104706 0.08104706]\n",
      " [0.10349726 0.10349726 0.10349726 ... 0.08655937 0.08655937 0.08655937]\n",
      " ...\n",
      " [0.09472247 0.09472247 0.09472247 ... 0.09190756 0.09190756 0.09190756]\n",
      " [0.09533982 0.09533982 0.09533982 ... 0.10539721 0.10539721 0.10539721]\n",
      " [0.08562803 0.08562803 0.08562803 ... 0.09929486 0.09929486 0.09929486]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.08737717 0.08737717 0.08737717 ... 0.08538679 0.08538679 0.08538679]\n",
      " [0.0906418  0.0906418  0.0906418  ... 0.09964925 0.09964925 0.09964925]\n",
      " [0.08544888 0.08544888 0.08544888 ... 0.09801837 0.09801837 0.09801837]\n",
      " ...\n",
      " [0.09678078 0.09678078 0.09678078 ... 0.08906878 0.08906878 0.08906878]\n",
      " [0.088358   0.088358   0.088358   ... 0.09481758 0.09481758 0.09481758]\n",
      " [0.0847678  0.0847678  0.0847678  ... 0.09385458 0.09385458 0.09385458]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.10372417 0.10372417 0.10372417 ... 0.09045094 0.09045094 0.09045094]\n",
      " [0.09100555 0.09100555 0.09100555 ... 0.09342661 0.09342661 0.09342661]\n",
      " [0.09319013 0.09319013 0.09319013 ... 0.07954691 0.07954691 0.07954691]\n",
      " ...\n",
      " [0.0823741  0.0823741  0.0823741  ... 0.08642127 0.08642127 0.08642127]\n",
      " [0.0945152  0.0945152  0.0945152  ... 0.10554124 0.10554124 0.10554124]\n",
      " [0.09336214 0.09336214 0.09336214 ... 0.07774585 0.07774585 0.07774585]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.08609451 0.08609451 0.08609451 ... 0.10513252 0.10513252 0.10513252]\n",
      " [0.08164602 0.08164602 0.08164602 ... 0.09390578 0.09390578 0.09390578]\n",
      " [0.08006752 0.08006752 0.08006752 ... 0.08845156 0.08845156 0.08845156]\n",
      " ...\n",
      " [0.09382664 0.09382664 0.09382664 ... 0.0882161  0.0882161  0.0882161 ]\n",
      " [0.08626813 0.08626813 0.08626813 ... 0.07996202 0.07996202 0.07996202]\n",
      " [0.09448674 0.09448674 0.09448674 ... 0.11940265 0.11940265 0.11940265]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1111  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.03466001 0.03466001 0.03466001 ... 0.03172823 0.03172823 0.03172823]\n",
      " [0.02698333 0.02698333 0.02698333 ... 0.02582636 0.02582636 0.02582636]\n",
      " [0.03620182 0.03620182 0.03620182 ... 0.0285269  0.0285269  0.0285269 ]\n",
      " ...\n",
      " [0.02879209 0.02879209 0.02879209 ... 0.02631722 0.02631722 0.02631722]\n",
      " [0.02682543 0.02682543 0.02682543 ... 0.03811241 0.03811241 0.03811241]\n",
      " [0.02005224 0.02005224 0.02005224 ... 0.01791499 0.01791499 0.01791499]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.02101266 0.02101266 0.02101266 ... 0.02764074 0.02764074 0.02764074]\n",
      " [0.03543362 0.03543362 0.03543362 ... 0.02559803 0.02559803 0.02559803]\n",
      " [0.030469   0.030469   0.030469   ... 0.04913303 0.04913303 0.04913303]\n",
      " ...\n",
      " [0.02746431 0.02746431 0.02746431 ... 0.01609766 0.01609766 0.01609766]\n",
      " [0.01632382 0.01632382 0.01632382 ... 0.03083614 0.03083614 0.03083614]\n",
      " [0.03645014 0.03645014 0.03645014 ... 0.01174016 0.01174016 0.01174016]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[ 0.02895535  0.02895535  0.02895535 ...  0.02547517  0.02547517\n",
      "   0.02547517]\n",
      " [ 0.01656999  0.01656999  0.01656999 ...  0.03109997  0.03109997\n",
      "   0.03109997]\n",
      " [ 0.02143458  0.02143458  0.02143458 ...  0.0032392   0.0032392\n",
      "   0.0032392 ]\n",
      " ...\n",
      " [ 0.02652271  0.02652271  0.02652271 ...  0.01802379  0.01802379\n",
      "   0.01802379]\n",
      " [ 0.0272274   0.0272274   0.0272274  ... -0.0048949  -0.0048949\n",
      "  -0.0048949 ]\n",
      " [ 0.03029206  0.03029206  0.03029206 ...  0.02105815  0.02105815\n",
      "   0.02105815]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.04337451 0.04337451 0.04337451 ... 0.02625773 0.02625773 0.02625773]\n",
      " [0.03945465 0.03945465 0.03945465 ... 0.01921081 0.01921081 0.01921081]\n",
      " [0.03774159 0.03774159 0.03774159 ... 0.02248256 0.02248256 0.02248256]\n",
      " ...\n",
      " [0.0374581  0.0374581  0.0374581  ... 0.02544012 0.02544012 0.02544012]\n",
      " [0.03355527 0.03355527 0.03355527 ... 0.02505317 0.02505317 0.02505317]\n",
      " [0.03376735 0.03376735 0.03376735 ... 0.02584489 0.02584489 0.02584489]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[0.0302929  0.0302929  0.0302929  ... 0.02049321 0.02049321 0.02049321]\n",
      " [0.04436296 0.04436296 0.04436296 ... 0.01528609 0.01528609 0.01528609]\n",
      " [0.0188646  0.0188646  0.0188646  ... 0.00630765 0.00630765 0.00630765]\n",
      " ...\n",
      " [0.02774002 0.02774002 0.02774002 ... 0.04341958 0.04341958 0.04341958]\n",
      " [0.04080764 0.04080764 0.04080764 ... 0.01698833 0.01698833 0.01698833]\n",
      " [0.01837436 0.01837436 0.01837436 ... 0.01320669 0.01320669 0.01320669]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[0.01848346 0.01848346 0.01848346 ... 0.03998875 0.03998875 0.03998875]\n",
      " [0.02254895 0.02254895 0.02254895 ... 0.04616692 0.04616692 0.04616692]\n",
      " [0.04080047 0.04080047 0.04080047 ... 0.03473735 0.03473735 0.03473735]\n",
      " ...\n",
      " [0.02774808 0.02774808 0.02774808 ... 0.04752131 0.04752131 0.04752131]\n",
      " [0.02246054 0.02246054 0.02246054 ... 0.03234085 0.03234085 0.03234085]\n",
      " [0.03275218 0.03275218 0.03275218 ... 0.01900406 0.01900406 0.01900406]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[0.02333616 0.02333616 0.02333616 ... 0.03057127 0.03057127 0.03057127]\n",
      " [0.02838733 0.02838733 0.02838733 ... 0.03974093 0.03974093 0.03974093]\n",
      " [0.03307772 0.03307772 0.03307772 ... 0.01668485 0.01668485 0.01668485]\n",
      " ...\n",
      " [0.0334579  0.0334579  0.0334579  ... 0.04867565 0.04867565 0.04867565]\n",
      " [0.04145489 0.04145489 0.04145489 ... 0.02592931 0.02592931 0.02592931]\n",
      " [0.02130611 0.02130611 0.02130611 ... 0.03312801 0.03312801 0.03312801]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[0.02592645 0.02592645 0.02592645 ... 0.02573388 0.02573388 0.02573388]\n",
      " [0.03147133 0.03147133 0.03147133 ... 0.0239864  0.0239864  0.0239864 ]\n",
      " [0.03322453 0.03322453 0.03322453 ... 0.0319016  0.0319016  0.0319016 ]\n",
      " ...\n",
      " [0.02461946 0.02461946 0.02461946 ... 0.03644301 0.03644301 0.03644301]\n",
      " [0.02268909 0.02268909 0.02268909 ... 0.03080148 0.03080148 0.03080148]\n",
      " [0.04984652 0.04984652 0.04984652 ... 0.02809234 0.02809234 0.02809234]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1212  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.02852174 -0.02852174 -0.02852174 ... -0.03548813 -0.03548813\n",
      "  -0.03548813]\n",
      " [-0.02429932 -0.02429932 -0.02429932 ... -0.04031744 -0.04031744\n",
      "  -0.04031744]\n",
      " [-0.0241817  -0.0241817  -0.0241817  ... -0.03916204 -0.03916204\n",
      "  -0.03916204]\n",
      " ...\n",
      " [-0.02424091 -0.02424091 -0.02424091 ... -0.04747197 -0.04747197\n",
      "  -0.04747197]\n",
      " [-0.00950959 -0.00950959 -0.00950959 ... -0.02470612 -0.02470612\n",
      "  -0.02470612]\n",
      " [-0.03828654 -0.03828654 -0.03828654 ... -0.04274574 -0.04274574\n",
      "  -0.04274574]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.02749066 -0.02749066 -0.02749066 ... -0.01580277 -0.01580277\n",
      "  -0.01580277]\n",
      " [-0.0385938  -0.0385938  -0.0385938  ... -0.04195129 -0.04195129\n",
      "  -0.04195129]\n",
      " [-0.03083159 -0.03083159 -0.03083159 ... -0.03222372 -0.03222372\n",
      "  -0.03222372]\n",
      " ...\n",
      " [-0.03123268 -0.03123268 -0.03123268 ... -0.03628759 -0.03628759\n",
      "  -0.03628759]\n",
      " [-0.02103107 -0.02103107 -0.02103107 ... -0.02561573 -0.02561573\n",
      "  -0.02561573]\n",
      " [-0.03731338 -0.03731338 -0.03731338 ... -0.02920656 -0.02920656\n",
      "  -0.02920656]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.05210156 -0.05210156 -0.05210156 ... -0.02483825 -0.02483825\n",
      "  -0.02483825]\n",
      " [-0.02111674 -0.02111674 -0.02111674 ... -0.02835035 -0.02835035\n",
      "  -0.02835035]\n",
      " [-0.03518111 -0.03518111 -0.03518111 ... -0.02370852 -0.02370852\n",
      "  -0.02370852]\n",
      " ...\n",
      " [-0.02746454 -0.02746454 -0.02746454 ... -0.02154805 -0.02154805\n",
      "  -0.02154805]\n",
      " [-0.04481836 -0.04481836 -0.04481836 ... -0.02189633 -0.02189633\n",
      "  -0.02189633]\n",
      " [-0.02446716 -0.02446716 -0.02446716 ... -0.02472494 -0.02472494\n",
      "  -0.02472494]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.02799388 -0.02799388 -0.02799388 ... -0.01750645 -0.01750645\n",
      "  -0.01750645]\n",
      " [-0.02747628 -0.02747628 -0.02747628 ... -0.04058114 -0.04058114\n",
      "  -0.04058114]\n",
      " [-0.04066295 -0.04066295 -0.04066295 ... -0.03092032 -0.03092032\n",
      "  -0.03092032]\n",
      " ...\n",
      " [-0.01865703 -0.01865703 -0.01865703 ... -0.02549174 -0.02549174\n",
      "  -0.02549174]\n",
      " [-0.02577283 -0.02577283 -0.02577283 ... -0.04187924 -0.04187924\n",
      "  -0.04187924]\n",
      " [-0.02923656 -0.02923656 -0.02923656 ... -0.03721638 -0.03721638\n",
      "  -0.03721638]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.03126866 -0.03126866 -0.03126866 ... -0.03134074 -0.03134074\n",
      "  -0.03134074]\n",
      " [-0.02751561 -0.02751561 -0.02751561 ... -0.03708369 -0.03708369\n",
      "  -0.03708369]\n",
      " [-0.0407884  -0.0407884  -0.0407884  ... -0.04872361 -0.04872361\n",
      "  -0.04872361]\n",
      " ...\n",
      " [-0.03105154 -0.03105154 -0.03105154 ... -0.01474717 -0.01474717\n",
      "  -0.01474717]\n",
      " [-0.02871628 -0.02871628 -0.02871628 ... -0.06141364 -0.06141364\n",
      "  -0.06141364]\n",
      " [-0.0453866  -0.0453866  -0.0453866  ... -0.05230027 -0.05230027\n",
      "  -0.05230027]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.03245939 -0.03245939 -0.03245939 ... -0.03913233 -0.03913233\n",
      "  -0.03913233]\n",
      " [-0.02579645 -0.02579645 -0.02579645 ... -0.04029911 -0.04029911\n",
      "  -0.04029911]\n",
      " [-0.03592315 -0.03592315 -0.03592315 ... -0.03873451 -0.03873451\n",
      "  -0.03873451]\n",
      " ...\n",
      " [-0.01905179 -0.01905179 -0.01905179 ... -0.02560707 -0.02560707\n",
      "  -0.02560707]\n",
      " [-0.02032471 -0.02032471 -0.02032471 ... -0.03271369 -0.03271369\n",
      "  -0.03271369]\n",
      " [-0.03547594 -0.03547594 -0.03547594 ... -0.02463401 -0.02463401\n",
      "  -0.02463401]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.03042594 -0.03042594 -0.03042594 ... -0.01031336 -0.01031336\n",
      "  -0.01031336]\n",
      " [-0.03250139 -0.03250139 -0.03250139 ... -0.03383121 -0.03383121\n",
      "  -0.03383121]\n",
      " [-0.03653418 -0.03653418 -0.03653418 ... -0.03320286 -0.03320286\n",
      "  -0.03320286]\n",
      " ...\n",
      " [-0.02393888 -0.02393888 -0.02393888 ... -0.02421683 -0.02421683\n",
      "  -0.02421683]\n",
      " [-0.04014727 -0.04014727 -0.04014727 ... -0.0350662  -0.0350662\n",
      "  -0.0350662 ]\n",
      " [-0.02142084 -0.02142084 -0.02142084 ... -0.02228149 -0.02228149\n",
      "  -0.02228149]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[ 0.01321973  0.01321973  0.01321973 ... -0.02296318 -0.02296318\n",
      "  -0.02296318]\n",
      " [-0.03812819 -0.03812819 -0.03812819 ... -0.02503513 -0.02503513\n",
      "  -0.02503513]\n",
      " [-0.0302411  -0.0302411  -0.0302411  ... -0.01785668 -0.01785668\n",
      "  -0.01785668]\n",
      " ...\n",
      " [-0.01244286 -0.01244286 -0.01244286 ... -0.03096524 -0.03096524\n",
      "  -0.03096524]\n",
      " [-0.04136307 -0.04136307 -0.04136307 ... -0.02920254 -0.02920254\n",
      "  -0.02920254]\n",
      " [-0.01356633 -0.01356633 -0.01356633 ... -0.01496297 -0.01496297\n",
      "  -0.01496297]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1313  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.08036695 -0.08036695 -0.08036695 ... -0.08348519 -0.08348519\n",
      "  -0.08348519]\n",
      " [-0.07493351 -0.07493351 -0.07493351 ... -0.06475803 -0.06475803\n",
      "  -0.06475803]\n",
      " [-0.08893535 -0.08893535 -0.08893535 ... -0.06772602 -0.06772602\n",
      "  -0.06772602]\n",
      " ...\n",
      " [-0.08709849 -0.08709849 -0.08709849 ... -0.08305312 -0.08305312\n",
      "  -0.08305312]\n",
      " [-0.08832006 -0.08832006 -0.08832006 ... -0.07618907 -0.07618907\n",
      "  -0.07618907]\n",
      " [-0.06172297 -0.06172297 -0.06172297 ... -0.05127793 -0.05127793\n",
      "  -0.05127793]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.10519496 -0.10519496 -0.10519496 ... -0.09352712 -0.09352712\n",
      "  -0.09352712]\n",
      " [-0.07540882 -0.07540882 -0.07540882 ... -0.07576916 -0.07576916\n",
      "  -0.07576916]\n",
      " [-0.08480813 -0.08480813 -0.08480813 ... -0.08554163 -0.08554163\n",
      "  -0.08554163]\n",
      " ...\n",
      " [-0.07967143 -0.07967143 -0.07967143 ... -0.07707937 -0.07707937\n",
      "  -0.07707937]\n",
      " [-0.08746275 -0.08746275 -0.08746275 ... -0.07927291 -0.07927291\n",
      "  -0.07927291]\n",
      " [-0.10301368 -0.10301368 -0.10301368 ... -0.08746895 -0.08746895\n",
      "  -0.08746895]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.06376095 -0.06376095 -0.06376095 ... -0.0713011  -0.0713011\n",
      "  -0.0713011 ]\n",
      " [-0.06954692 -0.06954692 -0.06954692 ... -0.06336594 -0.06336594\n",
      "  -0.06336594]\n",
      " [-0.07631945 -0.07631945 -0.07631945 ... -0.08218725 -0.08218725\n",
      "  -0.08218725]\n",
      " ...\n",
      " [-0.06806493 -0.06806493 -0.06806493 ... -0.08200409 -0.08200409\n",
      "  -0.08200409]\n",
      " [-0.05877829 -0.05877829 -0.05877829 ... -0.08165273 -0.08165273\n",
      "  -0.08165273]\n",
      " [-0.07733358 -0.07733358 -0.07733358 ... -0.09765474 -0.09765474\n",
      "  -0.09765474]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.08548216 -0.08548216 -0.08548216 ... -0.08414191 -0.08414191\n",
      "  -0.08414191]\n",
      " [-0.07657455 -0.07657455 -0.07657455 ... -0.09341283 -0.09341283\n",
      "  -0.09341283]\n",
      " [-0.07184414 -0.07184414 -0.07184414 ... -0.10633905 -0.10633905\n",
      "  -0.10633905]\n",
      " ...\n",
      " [-0.09270331 -0.09270331 -0.09270331 ... -0.08400293 -0.08400293\n",
      "  -0.08400293]\n",
      " [-0.085445   -0.085445   -0.085445   ... -0.08018094 -0.08018094\n",
      "  -0.08018094]\n",
      " [-0.09431108 -0.09431108 -0.09431108 ... -0.09268226 -0.09268226\n",
      "  -0.09268226]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.06653592 -0.06653592 -0.06653592 ... -0.1185369  -0.1185369\n",
      "  -0.1185369 ]\n",
      " [-0.09572838 -0.09572838 -0.09572838 ... -0.08109425 -0.08109425\n",
      "  -0.08109425]\n",
      " [-0.07465312 -0.07465312 -0.07465312 ... -0.09945889 -0.09945889\n",
      "  -0.09945889]\n",
      " ...\n",
      " [-0.078227   -0.078227   -0.078227   ... -0.07175725 -0.07175725\n",
      "  -0.07175725]\n",
      " [-0.07668497 -0.07668497 -0.07668497 ... -0.06878352 -0.06878352\n",
      "  -0.06878352]\n",
      " [-0.07446956 -0.07446956 -0.07446956 ... -0.08116236 -0.08116236\n",
      "  -0.08116236]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.07234639 -0.07234639 -0.07234639 ... -0.07671177 -0.07671177\n",
      "  -0.07671177]\n",
      " [-0.08538824 -0.08538824 -0.08538824 ... -0.0907922  -0.0907922\n",
      "  -0.0907922 ]\n",
      " [-0.08909117 -0.08909117 -0.08909117 ... -0.08744764 -0.08744764\n",
      "  -0.08744764]\n",
      " ...\n",
      " [-0.07560882 -0.07560882 -0.07560882 ... -0.08250045 -0.08250045\n",
      "  -0.08250045]\n",
      " [-0.06013396 -0.06013396 -0.06013396 ... -0.08935826 -0.08935826\n",
      "  -0.08935826]\n",
      " [-0.07774442 -0.07774442 -0.07774442 ... -0.0959533  -0.0959533\n",
      "  -0.0959533 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.09790263 -0.09790263 -0.09790263 ... -0.08894676 -0.08894676\n",
      "  -0.08894676]\n",
      " [-0.08709936 -0.08709936 -0.08709936 ... -0.09153657 -0.09153657\n",
      "  -0.09153657]\n",
      " [-0.08837379 -0.08837379 -0.08837379 ... -0.08753109 -0.08753109\n",
      "  -0.08753109]\n",
      " ...\n",
      " [-0.0861124  -0.0861124  -0.0861124  ... -0.07982577 -0.07982577\n",
      "  -0.07982577]\n",
      " [-0.08411132 -0.08411132 -0.08411132 ... -0.0836789  -0.0836789\n",
      "  -0.0836789 ]\n",
      " [-0.07589886 -0.07589886 -0.07589886 ... -0.08313043 -0.08313043\n",
      "  -0.08313043]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.0819213  -0.0819213  -0.0819213  ... -0.07179676 -0.07179676\n",
      "  -0.07179676]\n",
      " [-0.10415451 -0.10415451 -0.10415451 ... -0.07212503 -0.07212503\n",
      "  -0.07212503]\n",
      " [-0.06870851 -0.06870851 -0.06870851 ... -0.09257782 -0.09257782\n",
      "  -0.09257782]\n",
      " ...\n",
      " [-0.06651805 -0.06651805 -0.06651805 ... -0.0803348  -0.0803348\n",
      "  -0.0803348 ]\n",
      " [-0.10033591 -0.10033591 -0.10033591 ... -0.07739427 -0.07739427\n",
      "  -0.07739427]\n",
      " [-0.09364389 -0.09364389 -0.09364389 ... -0.08934171 -0.08934171\n",
      "  -0.08934171]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1414  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.17487223 -0.17487223 -0.17487223 ... -0.13533214 -0.13533214\n",
      "  -0.13533214]\n",
      " [-0.10775782 -0.10775782 -0.10775782 ... -0.13626972 -0.13626972\n",
      "  -0.13626972]\n",
      " [-0.12633008 -0.12633008 -0.12633008 ... -0.12529515 -0.12529515\n",
      "  -0.12529515]\n",
      " ...\n",
      " [-0.13451399 -0.13451399 -0.13451399 ... -0.13469103 -0.13469103\n",
      "  -0.13469103]\n",
      " [-0.1266084  -0.1266084  -0.1266084  ... -0.12638107 -0.12638107\n",
      "  -0.12638107]\n",
      " [-0.09949493 -0.09949493 -0.09949493 ... -0.11834907 -0.11834907\n",
      "  -0.11834907]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.14655013 -0.14655013 -0.14655013 ... -0.13033915 -0.13033915\n",
      "  -0.13033915]\n",
      " [-0.13314804 -0.13314804 -0.13314804 ... -0.13097164 -0.13097164\n",
      "  -0.13097164]\n",
      " [-0.12700425 -0.12700425 -0.12700425 ... -0.1542867  -0.1542867\n",
      "  -0.1542867 ]\n",
      " ...\n",
      " [-0.13218188 -0.13218188 -0.13218188 ... -0.12105897 -0.12105897\n",
      "  -0.12105897]\n",
      " [-0.12051008 -0.12051008 -0.12051008 ... -0.12972453 -0.12972453\n",
      "  -0.12972453]\n",
      " [-0.12716119 -0.12716119 -0.12716119 ... -0.13234545 -0.13234545\n",
      "  -0.13234545]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.12598577 -0.12598577 -0.12598577 ... -0.14143744 -0.14143744\n",
      "  -0.14143744]\n",
      " [-0.12265376 -0.12265376 -0.12265376 ... -0.12689437 -0.12689437\n",
      "  -0.12689437]\n",
      " [-0.11767863 -0.11767863 -0.11767863 ... -0.14731212 -0.14731212\n",
      "  -0.14731212]\n",
      " ...\n",
      " [-0.12603228 -0.12603228 -0.12603228 ... -0.11147492 -0.11147492\n",
      "  -0.11147492]\n",
      " [-0.1243452  -0.1243452  -0.1243452  ... -0.10748867 -0.10748867\n",
      "  -0.10748867]\n",
      " [-0.11438093 -0.11438093 -0.11438093 ... -0.15585653 -0.15585653\n",
      "  -0.15585653]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.11238346 -0.11238346 -0.11238346 ... -0.10541101 -0.10541101\n",
      "  -0.10541101]\n",
      " [-0.11097402 -0.11097402 -0.11097402 ... -0.13496685 -0.13496685\n",
      "  -0.13496685]\n",
      " [-0.13310085 -0.13310085 -0.13310085 ... -0.12415539 -0.12415539\n",
      "  -0.12415539]\n",
      " ...\n",
      " [-0.10668618 -0.10668618 -0.10668618 ... -0.14308032 -0.14308032\n",
      "  -0.14308032]\n",
      " [-0.13618489 -0.13618489 -0.13618489 ... -0.11940958 -0.11940958\n",
      "  -0.11940958]\n",
      " [-0.12419556 -0.12419556 -0.12419556 ... -0.12974441 -0.12974441\n",
      "  -0.12974441]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.11252594 -0.11252594 -0.11252594 ... -0.12579718 -0.12579718\n",
      "  -0.12579718]\n",
      " [-0.1192105  -0.1192105  -0.1192105  ... -0.1308043  -0.1308043\n",
      "  -0.1308043 ]\n",
      " [-0.12866615 -0.12866615 -0.12866615 ... -0.11974386 -0.11974386\n",
      "  -0.11974386]\n",
      " ...\n",
      " [-0.12223078 -0.12223078 -0.12223078 ... -0.12265706 -0.12265706\n",
      "  -0.12265706]\n",
      " [-0.1313988  -0.1313988  -0.1313988  ... -0.12434806 -0.12434806\n",
      "  -0.12434806]\n",
      " [-0.14779508 -0.14779508 -0.14779508 ... -0.11236013 -0.11236013\n",
      "  -0.11236013]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.14241144 -0.14241144 -0.14241144 ... -0.13149698 -0.13149698\n",
      "  -0.13149698]\n",
      " [-0.1594122  -0.1594122  -0.1594122  ... -0.09576359 -0.09576359\n",
      "  -0.09576359]\n",
      " [-0.11345179 -0.11345179 -0.11345179 ... -0.13191287 -0.13191287\n",
      "  -0.13191287]\n",
      " ...\n",
      " [-0.15396285 -0.15396285 -0.15396285 ... -0.13534078 -0.13534078\n",
      "  -0.13534078]\n",
      " [-0.14552562 -0.14552562 -0.14552562 ... -0.13196358 -0.13196358\n",
      "  -0.13196358]\n",
      " [-0.12925743 -0.12925743 -0.12925743 ... -0.13097507 -0.13097507\n",
      "  -0.13097507]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14087018 -0.14087018 -0.14087018 ... -0.15335089 -0.15335089\n",
      "  -0.15335089]\n",
      " [-0.13577113 -0.13577113 -0.13577113 ... -0.13187039 -0.13187039\n",
      "  -0.13187039]\n",
      " [-0.12527083 -0.12527083 -0.12527083 ... -0.12405115 -0.12405115\n",
      "  -0.12405115]\n",
      " ...\n",
      " [-0.13519871 -0.13519871 -0.13519871 ... -0.13054694 -0.13054694\n",
      "  -0.13054694]\n",
      " [-0.10167725 -0.10167725 -0.10167725 ... -0.11656007 -0.11656007\n",
      "  -0.11656007]\n",
      " [-0.13352352 -0.13352352 -0.13352352 ... -0.13033444 -0.13033444\n",
      "  -0.13033444]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.12743446 -0.12743446 -0.12743446 ... -0.12296075 -0.12296075\n",
      "  -0.12296075]\n",
      " [-0.1610833  -0.1610833  -0.1610833  ... -0.1455196  -0.1455196\n",
      "  -0.1455196 ]\n",
      " [-0.13493682 -0.13493682 -0.13493682 ... -0.12664492 -0.12664492\n",
      "  -0.12664492]\n",
      " ...\n",
      " [-0.14125788 -0.14125788 -0.14125788 ... -0.12557666 -0.12557666\n",
      "  -0.12557666]\n",
      " [-0.1169783  -0.1169783  -0.1169783  ... -0.10158609 -0.10158609\n",
      "  -0.10158609]\n",
      " [-0.13169949 -0.13169949 -0.13169949 ... -0.12069406 -0.12069406\n",
      "  -0.12069406]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1515  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.17034295 -0.17034295 -0.17034295 ... -0.17133328 -0.17133328\n",
      "  -0.17133328]\n",
      " [-0.1796367  -0.1796367  -0.1796367  ... -0.15954185 -0.15954185\n",
      "  -0.15954185]\n",
      " [-0.1740881  -0.1740881  -0.1740881  ... -0.15941298 -0.15941298\n",
      "  -0.15941298]\n",
      " ...\n",
      " [-0.18411662 -0.18411662 -0.18411662 ... -0.16710737 -0.16710737\n",
      "  -0.16710737]\n",
      " [-0.17084293 -0.17084293 -0.17084293 ... -0.1510309  -0.1510309\n",
      "  -0.1510309 ]\n",
      " [-0.15986866 -0.15986866 -0.15986866 ... -0.16997738 -0.16997738\n",
      "  -0.16997738]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1611843  -0.1611843  -0.1611843  ... -0.16402504 -0.16402504\n",
      "  -0.16402504]\n",
      " [-0.18507281 -0.18507281 -0.18507281 ... -0.19042152 -0.19042152\n",
      "  -0.19042152]\n",
      " [-0.19239463 -0.19239463 -0.19239463 ... -0.15362298 -0.15362298\n",
      "  -0.15362298]\n",
      " ...\n",
      " [-0.17838153 -0.17838153 -0.17838153 ... -0.15562415 -0.15562415\n",
      "  -0.15562415]\n",
      " [-0.18766782 -0.18766782 -0.18766782 ... -0.1694121  -0.1694121\n",
      "  -0.1694121 ]\n",
      " [-0.19547196 -0.19547196 -0.19547196 ... -0.16960096 -0.16960096\n",
      "  -0.16960096]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18155535 -0.18155535 -0.18155535 ... -0.185318   -0.185318\n",
      "  -0.185318  ]\n",
      " [-0.15128797 -0.15128797 -0.15128797 ... -0.12891424 -0.12891424\n",
      "  -0.12891424]\n",
      " [-0.16746327 -0.16746327 -0.16746327 ... -0.14089954 -0.14089954\n",
      "  -0.14089954]\n",
      " ...\n",
      " [-0.18295096 -0.18295096 -0.18295096 ... -0.17131087 -0.17131087\n",
      "  -0.17131087]\n",
      " [-0.20506826 -0.20506826 -0.20506826 ... -0.16861321 -0.16861321\n",
      "  -0.16861321]\n",
      " [-0.17355059 -0.17355059 -0.17355059 ... -0.18671247 -0.18671247\n",
      "  -0.18671247]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1795744  -0.1795744  -0.1795744  ... -0.15898262 -0.15898262\n",
      "  -0.15898262]\n",
      " [-0.16287455 -0.16287455 -0.16287455 ... -0.16229679 -0.16229679\n",
      "  -0.16229679]\n",
      " [-0.16437581 -0.16437581 -0.16437581 ... -0.20819399 -0.20819399\n",
      "  -0.20819399]\n",
      " ...\n",
      " [-0.16695017 -0.16695017 -0.16695017 ... -0.16674802 -0.16674802\n",
      "  -0.16674802]\n",
      " [-0.18111506 -0.18111506 -0.18111506 ... -0.16348228 -0.16348228\n",
      "  -0.16348228]\n",
      " [-0.17481083 -0.17481083 -0.17481083 ... -0.18095568 -0.18095568\n",
      "  -0.18095568]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.17438301 -0.17438301 -0.17438301 ... -0.1614462  -0.1614462\n",
      "  -0.1614462 ]\n",
      " [-0.13498756 -0.13498756 -0.13498756 ... -0.15987867 -0.15987867\n",
      "  -0.15987867]\n",
      " [-0.16098733 -0.16098733 -0.16098733 ... -0.1762209  -0.1762209\n",
      "  -0.1762209 ]\n",
      " ...\n",
      " [-0.15975636 -0.15975636 -0.15975636 ... -0.18280806 -0.18280806\n",
      "  -0.18280806]\n",
      " [-0.17923307 -0.17923307 -0.17923307 ... -0.17422616 -0.17422616\n",
      "  -0.17422616]\n",
      " [-0.16764215 -0.16764215 -0.16764215 ... -0.16731843 -0.16731843\n",
      "  -0.16731843]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16796468 -0.16796468 -0.16796468 ... -0.15330304 -0.15330304\n",
      "  -0.15330304]\n",
      " [-0.17735472 -0.17735472 -0.17735472 ... -0.17775394 -0.17775394\n",
      "  -0.17775394]\n",
      " [-0.170802   -0.170802   -0.170802   ... -0.18476021 -0.18476021\n",
      "  -0.18476021]\n",
      " ...\n",
      " [-0.17813    -0.17813    -0.17813    ... -0.16917023 -0.16917023\n",
      "  -0.16917023]\n",
      " [-0.18641786 -0.18641786 -0.18641786 ... -0.17381264 -0.17381264\n",
      "  -0.17381264]\n",
      " [-0.17883302 -0.17883302 -0.17883302 ... -0.1746487  -0.1746487\n",
      "  -0.1746487 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1745314  -0.1745314  -0.1745314  ... -0.17497844 -0.17497844\n",
      "  -0.17497844]\n",
      " [-0.17171633 -0.17171633 -0.17171633 ... -0.19295508 -0.19295508\n",
      "  -0.19295508]\n",
      " [-0.16720036 -0.16720036 -0.16720036 ... -0.16138846 -0.16138846\n",
      "  -0.16138846]\n",
      " ...\n",
      " [-0.16861111 -0.16861111 -0.16861111 ... -0.17386916 -0.17386916\n",
      "  -0.17386916]\n",
      " [-0.1795744  -0.1795744  -0.1795744  ... -0.16608359 -0.16608359\n",
      "  -0.16608359]\n",
      " [-0.16388305 -0.16388305 -0.16388305 ... -0.17409335 -0.17409335\n",
      "  -0.17409335]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15725575 -0.15725575 -0.15725575 ... -0.16407108 -0.16407108\n",
      "  -0.16407108]\n",
      " [-0.14597052 -0.14597052 -0.14597052 ... -0.17685796 -0.17685796\n",
      "  -0.17685796]\n",
      " [-0.16688606 -0.16688606 -0.16688606 ... -0.18189493 -0.18189493\n",
      "  -0.18189493]\n",
      " ...\n",
      " [-0.17592391 -0.17592391 -0.17592391 ... -0.16912453 -0.16912453\n",
      "  -0.16912453]\n",
      " [-0.19216993 -0.19216993 -0.19216993 ... -0.17397633 -0.17397633\n",
      "  -0.17397633]\n",
      " [-0.17107421 -0.17107421 -0.17107421 ... -0.17171891 -0.17171891\n",
      "  -0.17171891]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1616  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.21821567 -0.21821567 -0.21821567 ... -0.21728373 -0.21728373\n",
      "  -0.21728373]\n",
      " [-0.21300499 -0.21300499 -0.21300499 ... -0.2280635  -0.2280635\n",
      "  -0.2280635 ]\n",
      " [-0.2284374  -0.2284374  -0.2284374  ... -0.20640896 -0.20640896\n",
      "  -0.20640896]\n",
      " ...\n",
      " [-0.22416085 -0.22416085 -0.22416085 ... -0.19925462 -0.19925462\n",
      "  -0.19925462]\n",
      " [-0.20964347 -0.20964347 -0.20964347 ... -0.22059855 -0.22059855\n",
      "  -0.22059855]\n",
      " [-0.20121674 -0.20121674 -0.20121674 ... -0.21572433 -0.21572433\n",
      "  -0.21572433]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19584712 -0.19584712 -0.19584712 ... -0.20592439 -0.20592439\n",
      "  -0.20592439]\n",
      " [-0.21659112 -0.21659112 -0.21659112 ... -0.20824757 -0.20824757\n",
      "  -0.20824757]\n",
      " [-0.21707085 -0.21707085 -0.21707085 ... -0.21493559 -0.21493559\n",
      "  -0.21493559]\n",
      " ...\n",
      " [-0.20597196 -0.20597196 -0.20597196 ... -0.21226601 -0.21226601\n",
      "  -0.21226601]\n",
      " [-0.21448334 -0.21448334 -0.21448334 ... -0.22232367 -0.22232367\n",
      "  -0.22232367]\n",
      " [-0.22314714 -0.22314714 -0.22314714 ... -0.19758618 -0.19758618\n",
      "  -0.19758618]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20916174 -0.20916174 -0.20916174 ... -0.20235458 -0.20235458\n",
      "  -0.20235458]\n",
      " [-0.20964706 -0.20964706 -0.20964706 ... -0.19921729 -0.19921729\n",
      "  -0.19921729]\n",
      " [-0.21090987 -0.21090987 -0.21090987 ... -0.20636092 -0.20636092\n",
      "  -0.20636092]\n",
      " ...\n",
      " [-0.1974521  -0.1974521  -0.1974521  ... -0.2336465  -0.2336465\n",
      "  -0.2336465 ]\n",
      " [-0.20700513 -0.20700513 -0.20700513 ... -0.18486905 -0.18486905\n",
      "  -0.18486905]\n",
      " [-0.20448744 -0.20448744 -0.20448744 ... -0.2080696  -0.2080696\n",
      "  -0.2080696 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1996611  -0.1996611  -0.1996611  ... -0.21138012 -0.21138012\n",
      "  -0.21138012]\n",
      " [-0.22335002 -0.22335002 -0.22335002 ... -0.23856817 -0.23856817\n",
      "  -0.23856817]\n",
      " [-0.19355094 -0.19355094 -0.19355094 ... -0.22356014 -0.22356014\n",
      "  -0.22356014]\n",
      " ...\n",
      " [-0.17877997 -0.17877997 -0.17877997 ... -0.2103124  -0.2103124\n",
      "  -0.2103124 ]\n",
      " [-0.24041171 -0.24041171 -0.24041171 ... -0.21027279 -0.21027279\n",
      "  -0.21027279]\n",
      " [-0.22750352 -0.22750352 -0.22750352 ... -0.22710115 -0.22710115\n",
      "  -0.22710115]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.18557853 -0.18557853 -0.18557853 ... -0.2294093  -0.2294093\n",
      "  -0.2294093 ]\n",
      " [-0.19239408 -0.19239408 -0.19239408 ... -0.21053383 -0.21053383\n",
      "  -0.21053383]\n",
      " [-0.19028439 -0.19028439 -0.19028439 ... -0.21640424 -0.21640424\n",
      "  -0.21640424]\n",
      " ...\n",
      " [-0.20881002 -0.20881002 -0.20881002 ... -0.20857424 -0.20857424\n",
      "  -0.20857424]\n",
      " [-0.19312575 -0.19312575 -0.19312575 ... -0.1961278  -0.1961278\n",
      "  -0.1961278 ]\n",
      " [-0.22845505 -0.22845505 -0.22845505 ... -0.17724025 -0.17724025\n",
      "  -0.17724025]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.21108857 -0.21108857 -0.21108857 ... -0.21608493 -0.21608493\n",
      "  -0.21608493]\n",
      " [-0.17435628 -0.17435628 -0.17435628 ... -0.20663387 -0.20663387\n",
      "  -0.20663387]\n",
      " [-0.2124694  -0.2124694  -0.2124694  ... -0.21103959 -0.21103959\n",
      "  -0.21103959]\n",
      " ...\n",
      " [-0.21794951 -0.21794951 -0.21794951 ... -0.23521954 -0.23521954\n",
      "  -0.23521954]\n",
      " [-0.23233305 -0.23233305 -0.23233305 ... -0.19545047 -0.19545047\n",
      "  -0.19545047]\n",
      " [-0.19717279 -0.19717279 -0.19717279 ... -0.22398056 -0.22398056\n",
      "  -0.22398056]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21887371 -0.21887371 -0.21887371 ... -0.18569455 -0.18569455\n",
      "  -0.18569455]\n",
      " [-0.19929871 -0.19929871 -0.19929871 ... -0.18286668 -0.18286668\n",
      "  -0.18286668]\n",
      " [-0.21076746 -0.21076746 -0.21076746 ... -0.22823367 -0.22823367\n",
      "  -0.22823367]\n",
      " ...\n",
      " [-0.23386234 -0.23386234 -0.23386234 ... -0.20873725 -0.20873725\n",
      "  -0.20873725]\n",
      " [-0.21685961 -0.21685961 -0.21685961 ... -0.2228326  -0.2228326\n",
      "  -0.2228326 ]\n",
      " [-0.22420777 -0.22420777 -0.22420777 ... -0.18302695 -0.18302695\n",
      "  -0.18302695]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18969695 -0.18969695 -0.18969695 ... -0.22672658 -0.22672658\n",
      "  -0.22672658]\n",
      " [-0.2023178  -0.2023178  -0.2023178  ... -0.19691055 -0.19691055\n",
      "  -0.19691055]\n",
      " [-0.20481686 -0.20481686 -0.20481686 ... -0.22371396 -0.22371396\n",
      "  -0.22371396]\n",
      " ...\n",
      " [-0.21713546 -0.21713546 -0.21713546 ... -0.20725611 -0.20725611\n",
      "  -0.20725611]\n",
      " [-0.21114439 -0.21114439 -0.21114439 ... -0.22232893 -0.22232893\n",
      "  -0.22232893]\n",
      " [-0.22297029 -0.22297029 -0.22297029 ... -0.19477381 -0.19477381\n",
      "  -0.19477381]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1717  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23057221 -0.23057221 -0.23057221 ... -0.24310423 -0.24310423\n",
      "  -0.24310423]\n",
      " [-0.253322   -0.253322   -0.253322   ... -0.23602527 -0.23602527\n",
      "  -0.23602527]\n",
      " [-0.25718504 -0.25718504 -0.25718504 ... -0.252254   -0.252254\n",
      "  -0.252254  ]\n",
      " ...\n",
      " [-0.24635069 -0.24635069 -0.24635069 ... -0.24123359 -0.24123359\n",
      "  -0.24123359]\n",
      " [-0.23913334 -0.23913334 -0.23913334 ... -0.25966647 -0.25966647\n",
      "  -0.25966647]\n",
      " [-0.2684868  -0.2684868  -0.2684868  ... -0.25670537 -0.25670537\n",
      "  -0.25670537]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.25503194 -0.25503194 -0.25503194 ... -0.2373104  -0.2373104\n",
      "  -0.2373104 ]\n",
      " [-0.241189   -0.241189   -0.241189   ... -0.25546724 -0.25546724\n",
      "  -0.25546724]\n",
      " [-0.24621463 -0.24621463 -0.24621463 ... -0.25150746 -0.25150746\n",
      "  -0.25150746]\n",
      " ...\n",
      " [-0.22802228 -0.22802228 -0.22802228 ... -0.23546079 -0.23546079\n",
      "  -0.23546079]\n",
      " [-0.24654467 -0.24654467 -0.24654467 ... -0.26615342 -0.26615342\n",
      "  -0.26615342]\n",
      " [-0.26932883 -0.26932883 -0.26932883 ... -0.23249891 -0.23249891\n",
      "  -0.23249891]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20563175 -0.20563175 -0.20563175 ... -0.24036354 -0.24036354\n",
      "  -0.24036354]\n",
      " [-0.2579596  -0.2579596  -0.2579596  ... -0.25386572 -0.25386572\n",
      "  -0.25386572]\n",
      " [-0.24398181 -0.24398181 -0.24398181 ... -0.2488     -0.2488\n",
      "  -0.2488    ]\n",
      " ...\n",
      " [-0.2600992  -0.2600992  -0.2600992  ... -0.23481885 -0.23481885\n",
      "  -0.23481885]\n",
      " [-0.24071108 -0.24071108 -0.24071108 ... -0.27713287 -0.27713287\n",
      "  -0.27713287]\n",
      " [-0.23562425 -0.23562425 -0.23562425 ... -0.25067762 -0.25067762\n",
      "  -0.25067762]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24036627 -0.24036627 -0.24036627 ... -0.2498406  -0.2498406\n",
      "  -0.2498406 ]\n",
      " [-0.24335337 -0.24335337 -0.24335337 ... -0.24153064 -0.24153064\n",
      "  -0.24153064]\n",
      " [-0.22686256 -0.22686256 -0.22686256 ... -0.24473323 -0.24473323\n",
      "  -0.24473323]\n",
      " ...\n",
      " [-0.2444085  -0.2444085  -0.2444085  ... -0.25110382 -0.25110382\n",
      "  -0.25110382]\n",
      " [-0.25489846 -0.25489846 -0.25489846 ... -0.25825357 -0.25825357\n",
      "  -0.25825357]\n",
      " [-0.2528831  -0.2528831  -0.2528831  ... -0.25171793 -0.25171793\n",
      "  -0.25171793]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23730567 -0.23730567 -0.23730567 ... -0.25041166 -0.25041166\n",
      "  -0.25041166]\n",
      " [-0.24625853 -0.24625853 -0.24625853 ... -0.25779647 -0.25779647\n",
      "  -0.25779647]\n",
      " [-0.2618792  -0.2618792  -0.2618792  ... -0.24617702 -0.24617702\n",
      "  -0.24617702]\n",
      " ...\n",
      " [-0.24563085 -0.24563085 -0.24563085 ... -0.23699187 -0.23699187\n",
      "  -0.23699187]\n",
      " [-0.25686523 -0.25686523 -0.25686523 ... -0.2390075  -0.2390075\n",
      "  -0.2390075 ]\n",
      " [-0.23895821 -0.23895821 -0.23895821 ... -0.22843662 -0.22843662\n",
      "  -0.22843662]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24607071 -0.24607071 -0.24607071 ... -0.2642008  -0.2642008\n",
      "  -0.2642008 ]\n",
      " [-0.23320289 -0.23320289 -0.23320289 ... -0.23162225 -0.23162225\n",
      "  -0.23162225]\n",
      " [-0.25818795 -0.25818795 -0.25818795 ... -0.26274624 -0.26274624\n",
      "  -0.26274624]\n",
      " ...\n",
      " [-0.24232325 -0.24232325 -0.24232325 ... -0.25075564 -0.25075564\n",
      "  -0.25075564]\n",
      " [-0.25168025 -0.25168025 -0.25168025 ... -0.22609733 -0.22609733\n",
      "  -0.22609733]\n",
      " [-0.2583972  -0.2583972  -0.2583972  ... -0.2572524  -0.2572524\n",
      "  -0.2572524 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23986422 -0.23986422 -0.23986422 ... -0.24148272 -0.24148272\n",
      "  -0.24148272]\n",
      " [-0.241916   -0.241916   -0.241916   ... -0.25594234 -0.25594234\n",
      "  -0.25594234]\n",
      " [-0.2588527  -0.2588527  -0.2588527  ... -0.2420945  -0.2420945\n",
      "  -0.2420945 ]\n",
      " ...\n",
      " [-0.23201358 -0.23201358 -0.23201358 ... -0.2521652  -0.2521652\n",
      "  -0.2521652 ]\n",
      " [-0.2296447  -0.2296447  -0.2296447  ... -0.22552265 -0.22552265\n",
      "  -0.22552265]\n",
      " [-0.23412034 -0.23412034 -0.23412034 ... -0.23336282 -0.23336282\n",
      "  -0.23336282]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2539106  -0.2539106  -0.2539106  ... -0.23928493 -0.23928493\n",
      "  -0.23928493]\n",
      " [-0.22908671 -0.22908671 -0.22908671 ... -0.2545235  -0.2545235\n",
      "  -0.2545235 ]\n",
      " [-0.23124981 -0.23124981 -0.23124981 ... -0.2446034  -0.2446034\n",
      "  -0.2446034 ]\n",
      " ...\n",
      " [-0.2577439  -0.2577439  -0.2577439  ... -0.2784583  -0.2784583\n",
      "  -0.2784583 ]\n",
      " [-0.25706875 -0.25706875 -0.25706875 ... -0.25265586 -0.25265586\n",
      "  -0.25265586]\n",
      " [-0.23391145 -0.23391145 -0.23391145 ... -0.24585365 -0.24585365\n",
      "  -0.24585365]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1818  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.28378177 -0.28378177 -0.28378177 ... -0.27390888 -0.27390888\n",
      "  -0.27390888]\n",
      " [-0.2518457  -0.2518457  -0.2518457  ... -0.25641695 -0.25641695\n",
      "  -0.25641695]\n",
      " [-0.2901818  -0.2901818  -0.2901818  ... -0.26805726 -0.26805726\n",
      "  -0.26805726]\n",
      " ...\n",
      " [-0.26655582 -0.26655582 -0.26655582 ... -0.29064298 -0.29064298\n",
      "  -0.29064298]\n",
      " [-0.24405095 -0.24405095 -0.24405095 ... -0.27996662 -0.27996662\n",
      "  -0.27996662]\n",
      " [-0.28628623 -0.28628623 -0.28628623 ... -0.27485916 -0.27485916\n",
      "  -0.27485916]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.26861018 -0.26861018 -0.26861018 ... -0.2725684  -0.2725684\n",
      "  -0.2725684 ]\n",
      " [-0.2710703  -0.2710703  -0.2710703  ... -0.26980603 -0.26980603\n",
      "  -0.26980603]\n",
      " [-0.29338345 -0.29338345 -0.29338345 ... -0.27689803 -0.27689803\n",
      "  -0.27689803]\n",
      " ...\n",
      " [-0.24720074 -0.24720074 -0.24720074 ... -0.2791329  -0.2791329\n",
      "  -0.2791329 ]\n",
      " [-0.27353173 -0.27353173 -0.27353173 ... -0.30792657 -0.30792657\n",
      "  -0.30792657]\n",
      " [-0.2816649  -0.2816649  -0.2816649  ... -0.254487   -0.254487\n",
      "  -0.254487  ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.27659923 -0.27659923 -0.27659923 ... -0.2625646  -0.2625646\n",
      "  -0.2625646 ]\n",
      " [-0.27899528 -0.27899528 -0.27899528 ... -0.2853852  -0.2853852\n",
      "  -0.2853852 ]\n",
      " [-0.26963398 -0.26963398 -0.26963398 ... -0.2617389  -0.2617389\n",
      "  -0.2617389 ]\n",
      " ...\n",
      " [-0.26832348 -0.26832348 -0.26832348 ... -0.2501595  -0.2501595\n",
      "  -0.2501595 ]\n",
      " [-0.26837993 -0.26837993 -0.26837993 ... -0.28183842 -0.28183842\n",
      "  -0.28183842]\n",
      " [-0.26639137 -0.26639137 -0.26639137 ... -0.2911839  -0.2911839\n",
      "  -0.2911839 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.26263815 -0.26263815 -0.26263815 ... -0.2341427  -0.2341427\n",
      "  -0.2341427 ]\n",
      " [-0.30969337 -0.30969337 -0.30969337 ... -0.2688436  -0.2688436\n",
      "  -0.2688436 ]\n",
      " [-0.26999807 -0.26999807 -0.26999807 ... -0.26476222 -0.26476222\n",
      "  -0.26476222]\n",
      " ...\n",
      " [-0.25358373 -0.25358373 -0.25358373 ... -0.26177627 -0.26177627\n",
      "  -0.26177627]\n",
      " [-0.27928942 -0.27928942 -0.27928942 ... -0.278804   -0.278804\n",
      "  -0.278804  ]\n",
      " [-0.3028634  -0.3028634  -0.3028634  ... -0.27395043 -0.27395043\n",
      "  -0.27395043]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.25776958 -0.25776958 -0.25776958 ... -0.2531976  -0.2531976\n",
      "  -0.2531976 ]\n",
      " [-0.25639272 -0.25639272 -0.25639272 ... -0.26465923 -0.26465923\n",
      "  -0.26465923]\n",
      " [-0.26687717 -0.26687717 -0.26687717 ... -0.26470613 -0.26470613\n",
      "  -0.26470613]\n",
      " ...\n",
      " [-0.28487194 -0.28487194 -0.28487194 ... -0.2820628  -0.2820628\n",
      "  -0.2820628 ]\n",
      " [-0.25619847 -0.25619847 -0.25619847 ... -0.2687176  -0.2687176\n",
      "  -0.2687176 ]\n",
      " [-0.29643747 -0.29643747 -0.29643747 ... -0.3000363  -0.3000363\n",
      "  -0.3000363 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.3045588  -0.3045588  -0.3045588  ... -0.26813287 -0.26813287\n",
      "  -0.26813287]\n",
      " [-0.27863112 -0.27863112 -0.27863112 ... -0.273818   -0.273818\n",
      "  -0.273818  ]\n",
      " [-0.27770215 -0.27770215 -0.27770215 ... -0.27316087 -0.27316087\n",
      "  -0.27316087]\n",
      " ...\n",
      " [-0.27075255 -0.27075255 -0.27075255 ... -0.27963242 -0.27963242\n",
      "  -0.27963242]\n",
      " [-0.26409572 -0.26409572 -0.26409572 ... -0.3014991  -0.3014991\n",
      "  -0.3014991 ]\n",
      " [-0.26489216 -0.26489216 -0.26489216 ... -0.29010373 -0.29010373\n",
      "  -0.29010373]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2532282  -0.2532282  -0.2532282  ... -0.29261005 -0.29261005\n",
      "  -0.29261005]\n",
      " [-0.29190302 -0.29190302 -0.29190302 ... -0.25137788 -0.25137788\n",
      "  -0.25137788]\n",
      " [-0.29877418 -0.29877418 -0.29877418 ... -0.2512166  -0.2512166\n",
      "  -0.2512166 ]\n",
      " ...\n",
      " [-0.26245174 -0.26245174 -0.26245174 ... -0.26680988 -0.26680988\n",
      "  -0.26680988]\n",
      " [-0.2691456  -0.2691456  -0.2691456  ... -0.24677    -0.24677\n",
      "  -0.24677   ]\n",
      " [-0.27240795 -0.27240795 -0.27240795 ... -0.27131796 -0.27131796\n",
      "  -0.27131796]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2893701  -0.2893701  -0.2893701  ... -0.2744174  -0.2744174\n",
      "  -0.2744174 ]\n",
      " [-0.26395565 -0.26395565 -0.26395565 ... -0.27849576 -0.27849576\n",
      "  -0.27849576]\n",
      " [-0.28116703 -0.28116703 -0.28116703 ... -0.2760107  -0.2760107\n",
      "  -0.2760107 ]\n",
      " ...\n",
      " [-0.28218758 -0.28218758 -0.28218758 ... -0.27634722 -0.27634722\n",
      "  -0.27634722]\n",
      " [-0.25285238 -0.25285238 -0.25285238 ... -0.24106067 -0.24106067\n",
      "  -0.24106067]\n",
      " [-0.26568672 -0.26568672 -0.26568672 ... -0.2536044  -0.2536044\n",
      "  -0.2536044 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   1919  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2797534  -0.2797534  -0.2797534  ... -0.27815503 -0.27815503\n",
      "  -0.27815503]\n",
      " [-0.29525018 -0.29525018 -0.29525018 ... -0.3032462  -0.3032462\n",
      "  -0.3032462 ]\n",
      " [-0.27514595 -0.27514595 -0.27514595 ... -0.28187245 -0.28187245\n",
      "  -0.28187245]\n",
      " ...\n",
      " [-0.28432584 -0.28432584 -0.28432584 ... -0.28062695 -0.28062695\n",
      "  -0.28062695]\n",
      " [-0.296606   -0.296606   -0.296606   ... -0.31403136 -0.31403136\n",
      "  -0.31403136]\n",
      " [-0.29528973 -0.29528973 -0.29528973 ... -0.2861325  -0.2861325\n",
      "  -0.2861325 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.32181713 -0.32181713 -0.32181713 ... -0.29737788 -0.29737788\n",
      "  -0.29737788]\n",
      " [-0.2892059  -0.2892059  -0.2892059  ... -0.29361174 -0.29361174\n",
      "  -0.29361174]\n",
      " [-0.29315072 -0.29315072 -0.29315072 ... -0.30869493 -0.30869493\n",
      "  -0.30869493]\n",
      " ...\n",
      " [-0.28053385 -0.28053385 -0.28053385 ... -0.29473233 -0.29473233\n",
      "  -0.29473233]\n",
      " [-0.28540596 -0.28540596 -0.28540596 ... -0.2936319  -0.2936319\n",
      "  -0.2936319 ]\n",
      " [-0.30036208 -0.30036208 -0.30036208 ... -0.29702425 -0.29702425\n",
      "  -0.29702425]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.30812743 -0.30812743 -0.30812743 ... -0.33037218 -0.33037218\n",
      "  -0.33037218]\n",
      " [-0.28817254 -0.28817254 -0.28817254 ... -0.28671163 -0.28671163\n",
      "  -0.28671163]\n",
      " [-0.30242634 -0.30242634 -0.30242634 ... -0.30997562 -0.30997562\n",
      "  -0.30997562]\n",
      " ...\n",
      " [-0.29145706 -0.29145706 -0.29145706 ... -0.28260267 -0.28260267\n",
      "  -0.28260267]\n",
      " [-0.2797789  -0.2797789  -0.2797789  ... -0.28468162 -0.28468162\n",
      "  -0.28468162]\n",
      " [-0.28278536 -0.28278536 -0.28278536 ... -0.2528563  -0.2528563\n",
      "  -0.2528563 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2819752  -0.2819752  -0.2819752  ... -0.30500767 -0.30500767\n",
      "  -0.30500767]\n",
      " [-0.2996728  -0.2996728  -0.2996728  ... -0.2861158  -0.2861158\n",
      "  -0.2861158 ]\n",
      " [-0.2867563  -0.2867563  -0.2867563  ... -0.2988876  -0.2988876\n",
      "  -0.2988876 ]\n",
      " ...\n",
      " [-0.30879116 -0.30879116 -0.30879116 ... -0.29969236 -0.29969236\n",
      "  -0.29969236]\n",
      " [-0.28310788 -0.28310788 -0.28310788 ... -0.28076214 -0.28076214\n",
      "  -0.28076214]\n",
      " [-0.34134018 -0.34134018 -0.34134018 ... -0.32739896 -0.32739896\n",
      "  -0.32739896]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.28960684 -0.28960684 -0.28960684 ... -0.28125203 -0.28125203\n",
      "  -0.28125203]\n",
      " [-0.3064102  -0.3064102  -0.3064102  ... -0.30044872 -0.30044872\n",
      "  -0.30044872]\n",
      " [-0.29099596 -0.29099596 -0.29099596 ... -0.2605029  -0.2605029\n",
      "  -0.2605029 ]\n",
      " ...\n",
      " [-0.29638803 -0.29638803 -0.29638803 ... -0.3035868  -0.3035868\n",
      "  -0.3035868 ]\n",
      " [-0.29624936 -0.29624936 -0.29624936 ... -0.29819614 -0.29819614\n",
      "  -0.29819614]\n",
      " [-0.2994034  -0.2994034  -0.2994034  ... -0.28842664 -0.28842664\n",
      "  -0.28842664]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.3145328  -0.3145328  -0.3145328  ... -0.2750369  -0.2750369\n",
      "  -0.2750369 ]\n",
      " [-0.28332084 -0.28332084 -0.28332084 ... -0.33094484 -0.33094484\n",
      "  -0.33094484]\n",
      " [-0.3021652  -0.3021652  -0.3021652  ... -0.28959057 -0.28959057\n",
      "  -0.28959057]\n",
      " ...\n",
      " [-0.3157429  -0.3157429  -0.3157429  ... -0.27956775 -0.27956775\n",
      "  -0.27956775]\n",
      " [-0.29254475 -0.29254475 -0.29254475 ... -0.30029994 -0.30029994\n",
      "  -0.30029994]\n",
      " [-0.27229038 -0.27229038 -0.27229038 ... -0.30886158 -0.30886158\n",
      "  -0.30886158]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.30756545 -0.30756545 -0.30756545 ... -0.2570176  -0.2570176\n",
      "  -0.2570176 ]\n",
      " [-0.29633275 -0.29633275 -0.29633275 ... -0.3099647  -0.3099647\n",
      "  -0.3099647 ]\n",
      " [-0.29539108 -0.29539108 -0.29539108 ... -0.30012643 -0.30012643\n",
      "  -0.30012643]\n",
      " ...\n",
      " [-0.30277866 -0.30277866 -0.30277866 ... -0.28505564 -0.28505564\n",
      "  -0.28505564]\n",
      " [-0.3036524  -0.3036524  -0.3036524  ... -0.3309218  -0.3309218\n",
      "  -0.3309218 ]\n",
      " [-0.27706498 -0.27706498 -0.27706498 ... -0.29898155 -0.29898155\n",
      "  -0.29898155]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.28304434 -0.28304434 -0.28304434 ... -0.28734958 -0.28734958\n",
      "  -0.28734958]\n",
      " [-0.30845338 -0.30845338 -0.30845338 ... -0.30182338 -0.30182338\n",
      "  -0.30182338]\n",
      " [-0.28692037 -0.28692037 -0.28692037 ... -0.29562625 -0.29562625\n",
      "  -0.29562625]\n",
      " ...\n",
      " [-0.30988312 -0.30988312 -0.30988312 ... -0.3074304  -0.3074304\n",
      "  -0.3074304 ]\n",
      " [-0.28374046 -0.28374046 -0.28374046 ... -0.2746107  -0.2746107\n",
      "  -0.2746107 ]\n",
      " [-0.31488702 -0.31488702 -0.31488702 ... -0.29483104 -0.29483104\n",
      "  -0.29483104]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2020  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.3247959  -0.3247959  -0.3247959  ... -0.27718997 -0.27718997\n",
      "  -0.27718997]\n",
      " [-0.29558674 -0.29558674 -0.29558674 ... -0.31112006 -0.31112006\n",
      "  -0.31112006]\n",
      " [-0.3060099  -0.3060099  -0.3060099  ... -0.30911535 -0.30911535\n",
      "  -0.30911535]\n",
      " ...\n",
      " [-0.31649962 -0.31649962 -0.31649962 ... -0.3106779  -0.3106779\n",
      "  -0.3106779 ]\n",
      " [-0.31722373 -0.31722373 -0.31722373 ... -0.31988508 -0.31988508\n",
      "  -0.31988508]\n",
      " [-0.271111   -0.271111   -0.271111   ... -0.30045372 -0.30045372\n",
      "  -0.30045372]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.30113983 -0.30113983 -0.30113983 ... -0.30418557 -0.30418557\n",
      "  -0.30418557]\n",
      " [-0.3022943  -0.3022943  -0.3022943  ... -0.3086636  -0.3086636\n",
      "  -0.3086636 ]\n",
      " [-0.3154026  -0.3154026  -0.3154026  ... -0.31364048 -0.31364048\n",
      "  -0.31364048]\n",
      " ...\n",
      " [-0.28788632 -0.28788632 -0.28788632 ... -0.29315567 -0.29315567\n",
      "  -0.29315567]\n",
      " [-0.2624233  -0.2624233  -0.2624233  ... -0.30738226 -0.30738226\n",
      "  -0.30738226]\n",
      " [-0.31501353 -0.31501353 -0.31501353 ... -0.27239022 -0.27239022\n",
      "  -0.27239022]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28261223 -0.28261223 -0.28261223 ... -0.30475467 -0.30475467\n",
      "  -0.30475467]\n",
      " [-0.29271868 -0.29271868 -0.29271868 ... -0.29820237 -0.29820237\n",
      "  -0.29820237]\n",
      " [-0.26265275 -0.26265275 -0.26265275 ... -0.30924162 -0.30924162\n",
      "  -0.30924162]\n",
      " ...\n",
      " [-0.30109704 -0.30109704 -0.30109704 ... -0.29440108 -0.29440108\n",
      "  -0.29440108]\n",
      " [-0.3308077  -0.3308077  -0.3308077  ... -0.28151596 -0.28151596\n",
      "  -0.28151596]\n",
      " [-0.29967368 -0.29967368 -0.29967368 ... -0.3095507  -0.3095507\n",
      "  -0.3095507 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.29831678 -0.29831678 -0.29831678 ... -0.30967456 -0.30967456\n",
      "  -0.30967456]\n",
      " [-0.31055725 -0.31055725 -0.31055725 ... -0.31774244 -0.31774244\n",
      "  -0.31774244]\n",
      " [-0.3157764  -0.3157764  -0.3157764  ... -0.27433664 -0.27433664\n",
      "  -0.27433664]\n",
      " ...\n",
      " [-0.31843233 -0.31843233 -0.31843233 ... -0.30749777 -0.30749777\n",
      "  -0.30749777]\n",
      " [-0.321081   -0.321081   -0.321081   ... -0.28974333 -0.28974333\n",
      "  -0.28974333]\n",
      " [-0.32367277 -0.32367277 -0.32367277 ... -0.27158654 -0.27158654\n",
      "  -0.27158654]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.30004576 -0.30004576 -0.30004576 ... -0.30129188 -0.30129188\n",
      "  -0.30129188]\n",
      " [-0.27407452 -0.27407452 -0.27407452 ... -0.29746908 -0.29746908\n",
      "  -0.29746908]\n",
      " [-0.32805672 -0.32805672 -0.32805672 ... -0.3282205  -0.3282205\n",
      "  -0.3282205 ]\n",
      " ...\n",
      " [-0.2584542  -0.2584542  -0.2584542  ... -0.30776444 -0.30776444\n",
      "  -0.30776444]\n",
      " [-0.30064857 -0.30064857 -0.30064857 ... -0.30684966 -0.30684966\n",
      "  -0.30684966]\n",
      " [-0.29471582 -0.29471582 -0.29471582 ... -0.2930435  -0.2930435\n",
      "  -0.2930435 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.28290823 -0.28290823 -0.28290823 ... -0.3102165  -0.3102165\n",
      "  -0.3102165 ]\n",
      " [-0.3113042  -0.3113042  -0.3113042  ... -0.28651857 -0.28651857\n",
      "  -0.28651857]\n",
      " [-0.31294394 -0.31294394 -0.31294394 ... -0.30894366 -0.30894366\n",
      "  -0.30894366]\n",
      " ...\n",
      " [-0.3335798  -0.3335798  -0.3335798  ... -0.30469698 -0.30469698\n",
      "  -0.30469698]\n",
      " [-0.30331498 -0.30331498 -0.30331498 ... -0.32693917 -0.32693917\n",
      "  -0.32693917]\n",
      " [-0.33106196 -0.33106196 -0.33106196 ... -0.32021624 -0.32021624\n",
      "  -0.32021624]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28648034 -0.28648034 -0.28648034 ... -0.2864005  -0.2864005\n",
      "  -0.2864005 ]\n",
      " [-0.2770051  -0.2770051  -0.2770051  ... -0.2959671  -0.2959671\n",
      "  -0.2959671 ]\n",
      " [-0.31885952 -0.31885952 -0.31885952 ... -0.31448793 -0.31448793\n",
      "  -0.31448793]\n",
      " ...\n",
      " [-0.2812934  -0.2812934  -0.2812934  ... -0.29454935 -0.29454935\n",
      "  -0.29454935]\n",
      " [-0.31417572 -0.31417572 -0.31417572 ... -0.31781125 -0.31781125\n",
      "  -0.31781125]\n",
      " [-0.31561953 -0.31561953 -0.31561953 ... -0.30614847 -0.30614847\n",
      "  -0.30614847]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.3161555  -0.3161555  -0.3161555  ... -0.30208832 -0.30208832\n",
      "  -0.30208832]\n",
      " [-0.31487444 -0.31487444 -0.31487444 ... -0.3022612  -0.3022612\n",
      "  -0.3022612 ]\n",
      " [-0.30111253 -0.30111253 -0.30111253 ... -0.26954857 -0.26954857\n",
      "  -0.26954857]\n",
      " ...\n",
      " [-0.31705892 -0.31705892 -0.31705892 ... -0.3087765  -0.3087765\n",
      "  -0.3087765 ]\n",
      " [-0.29052728 -0.29052728 -0.29052728 ... -0.27886248 -0.27886248\n",
      "  -0.27886248]\n",
      " [-0.29038456 -0.29038456 -0.29038456 ... -0.30555174 -0.30555174\n",
      "  -0.30555174]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2121  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.29434955 -0.29434955 -0.29434955 ... -0.3288989  -0.3288989\n",
      "  -0.3288989 ]\n",
      " [-0.31581306 -0.31581306 -0.31581306 ... -0.3123786  -0.3123786\n",
      "  -0.3123786 ]\n",
      " [-0.29792657 -0.29792657 -0.29792657 ... -0.31096    -0.31096\n",
      "  -0.31096   ]\n",
      " ...\n",
      " [-0.30340093 -0.30340093 -0.30340093 ... -0.28990504 -0.28990504\n",
      "  -0.28990504]\n",
      " [-0.308826   -0.308826   -0.308826   ... -0.30395693 -0.30395693\n",
      "  -0.30395693]\n",
      " [-0.29838538 -0.29838538 -0.29838538 ... -0.3046275  -0.3046275\n",
      "  -0.3046275 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.3137595  -0.3137595  -0.3137595  ... -0.34154692 -0.34154692\n",
      "  -0.34154692]\n",
      " [-0.3084019  -0.3084019  -0.3084019  ... -0.28463668 -0.28463668\n",
      "  -0.28463668]\n",
      " [-0.2836891  -0.2836891  -0.2836891  ... -0.3070513  -0.3070513\n",
      "  -0.3070513 ]\n",
      " ...\n",
      " [-0.31498265 -0.31498265 -0.31498265 ... -0.3088838  -0.3088838\n",
      "  -0.3088838 ]\n",
      " [-0.29942122 -0.29942122 -0.29942122 ... -0.30515352 -0.30515352\n",
      "  -0.30515352]\n",
      " [-0.28876895 -0.28876895 -0.28876895 ... -0.30965635 -0.30965635\n",
      "  -0.30965635]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.3094587  -0.3094587  -0.3094587  ... -0.30968165 -0.30968165\n",
      "  -0.30968165]\n",
      " [-0.3283684  -0.3283684  -0.3283684  ... -0.29884627 -0.29884627\n",
      "  -0.29884627]\n",
      " [-0.3014267  -0.3014267  -0.3014267  ... -0.3228361  -0.3228361\n",
      "  -0.3228361 ]\n",
      " ...\n",
      " [-0.2836056  -0.2836056  -0.2836056  ... -0.27962512 -0.27962512\n",
      "  -0.27962512]\n",
      " [-0.31022775 -0.31022775 -0.31022775 ... -0.3114977  -0.3114977\n",
      "  -0.3114977 ]\n",
      " [-0.30297363 -0.30297363 -0.30297363 ... -0.29791602 -0.29791602\n",
      "  -0.29791602]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2996969  -0.2996969  -0.2996969  ... -0.31619436 -0.31619436\n",
      "  -0.31619436]\n",
      " [-0.3043476  -0.3043476  -0.3043476  ... -0.3079921  -0.3079921\n",
      "  -0.3079921 ]\n",
      " [-0.32044995 -0.32044995 -0.32044995 ... -0.33852828 -0.33852828\n",
      "  -0.33852828]\n",
      " ...\n",
      " [-0.3146191  -0.3146191  -0.3146191  ... -0.3337138  -0.3337138\n",
      "  -0.3337138 ]\n",
      " [-0.30985218 -0.30985218 -0.30985218 ... -0.30149257 -0.30149257\n",
      "  -0.30149257]\n",
      " [-0.30994368 -0.30994368 -0.30994368 ... -0.3378619  -0.3378619\n",
      "  -0.3378619 ]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.319009   -0.319009   -0.319009   ... -0.3160634  -0.3160634\n",
      "  -0.3160634 ]\n",
      " [-0.3031947  -0.3031947  -0.3031947  ... -0.29683983 -0.29683983\n",
      "  -0.29683983]\n",
      " [-0.30938685 -0.30938685 -0.30938685 ... -0.3269104  -0.3269104\n",
      "  -0.3269104 ]\n",
      " ...\n",
      " [-0.32052308 -0.32052308 -0.32052308 ... -0.30940235 -0.30940235\n",
      "  -0.30940235]\n",
      " [-0.29752505 -0.29752505 -0.29752505 ... -0.3158247  -0.3158247\n",
      "  -0.3158247 ]\n",
      " [-0.30363798 -0.30363798 -0.30363798 ... -0.32834294 -0.32834294\n",
      "  -0.32834294]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.31050515 -0.31050515 -0.31050515 ... -0.3009111  -0.3009111\n",
      "  -0.3009111 ]\n",
      " [-0.3012192  -0.3012192  -0.3012192  ... -0.31445804 -0.31445804\n",
      "  -0.31445804]\n",
      " [-0.25030798 -0.25030798 -0.25030798 ... -0.33975744 -0.33975744\n",
      "  -0.33975744]\n",
      " ...\n",
      " [-0.29498863 -0.29498863 -0.29498863 ... -0.31542495 -0.31542495\n",
      "  -0.31542495]\n",
      " [-0.2987432  -0.2987432  -0.2987432  ... -0.31379437 -0.31379437\n",
      "  -0.31379437]\n",
      " [-0.32711312 -0.32711312 -0.32711312 ... -0.31240535 -0.31240535\n",
      "  -0.31240535]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.33177313 -0.33177313 -0.33177313 ... -0.29177758 -0.29177758\n",
      "  -0.29177758]\n",
      " [-0.31262612 -0.31262612 -0.31262612 ... -0.31074163 -0.31074163\n",
      "  -0.31074163]\n",
      " [-0.26922682 -0.26922682 -0.26922682 ... -0.32754964 -0.32754964\n",
      "  -0.32754964]\n",
      " ...\n",
      " [-0.2986963  -0.2986963  -0.2986963  ... -0.31963903 -0.31963903\n",
      "  -0.31963903]\n",
      " [-0.2866307  -0.2866307  -0.2866307  ... -0.3155861  -0.3155861\n",
      "  -0.3155861 ]\n",
      " [-0.31161118 -0.31161118 -0.31161118 ... -0.30134034 -0.30134034\n",
      "  -0.30134034]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.31392285 -0.31392285 -0.31392285 ... -0.29388607 -0.29388607\n",
      "  -0.29388607]\n",
      " [-0.2797379  -0.2797379  -0.2797379  ... -0.3026418  -0.3026418\n",
      "  -0.3026418 ]\n",
      " [-0.31173792 -0.31173792 -0.31173792 ... -0.28673857 -0.28673857\n",
      "  -0.28673857]\n",
      " ...\n",
      " [-0.31705254 -0.31705254 -0.31705254 ... -0.31438088 -0.31438088\n",
      "  -0.31438088]\n",
      " [-0.30384725 -0.30384725 -0.30384725 ... -0.31497106 -0.31497106\n",
      "  -0.31497106]\n",
      " [-0.32072246 -0.32072246 -0.32072246 ... -0.3046922  -0.3046922\n",
      "  -0.3046922 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2222  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.29779014 -0.29779014 -0.29779014 ... -0.31736487 -0.31736487\n",
      "  -0.31736487]\n",
      " [-0.30566064 -0.30566064 -0.30566064 ... -0.28571618 -0.28571618\n",
      "  -0.28571618]\n",
      " [-0.30899292 -0.30899292 -0.30899292 ... -0.29099858 -0.29099858\n",
      "  -0.29099858]\n",
      " ...\n",
      " [-0.27110368 -0.27110368 -0.27110368 ... -0.31589392 -0.31589392\n",
      "  -0.31589392]\n",
      " [-0.28874755 -0.28874755 -0.28874755 ... -0.3113771  -0.3113771\n",
      "  -0.3113771 ]\n",
      " [-0.31094235 -0.31094235 -0.31094235 ... -0.30825585 -0.30825585\n",
      "  -0.30825585]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.30907494 -0.30907494 -0.30907494 ... -0.31867325 -0.31867325\n",
      "  -0.31867325]\n",
      " [-0.2984959  -0.2984959  -0.2984959  ... -0.3099621  -0.3099621\n",
      "  -0.3099621 ]\n",
      " [-0.31564236 -0.31564236 -0.31564236 ... -0.33370316 -0.33370316\n",
      "  -0.33370316]\n",
      " ...\n",
      " [-0.29082823 -0.29082823 -0.29082823 ... -0.28257868 -0.28257868\n",
      "  -0.28257868]\n",
      " [-0.26778004 -0.26778004 -0.26778004 ... -0.30836046 -0.30836046\n",
      "  -0.30836046]\n",
      " [-0.28235435 -0.28235435 -0.28235435 ... -0.30324095 -0.30324095\n",
      "  -0.30324095]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.32858354 -0.32858354 -0.32858354 ... -0.29953814 -0.29953814\n",
      "  -0.29953814]\n",
      " [-0.31220812 -0.31220812 -0.31220812 ... -0.3207736  -0.3207736\n",
      "  -0.3207736 ]\n",
      " [-0.3098514  -0.3098514  -0.3098514  ... -0.32065862 -0.32065862\n",
      "  -0.32065862]\n",
      " ...\n",
      " [-0.27782655 -0.27782655 -0.27782655 ... -0.35292795 -0.35292795\n",
      "  -0.35292795]\n",
      " [-0.2845305  -0.2845305  -0.2845305  ... -0.3035884  -0.3035884\n",
      "  -0.3035884 ]\n",
      " [-0.30100843 -0.30100843 -0.30100843 ... -0.29402834 -0.29402834\n",
      "  -0.29402834]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.27961528 -0.27961528 -0.27961528 ... -0.31082496 -0.31082496\n",
      "  -0.31082496]\n",
      " [-0.3002985  -0.3002985  -0.3002985  ... -0.3096675  -0.3096675\n",
      "  -0.3096675 ]\n",
      " [-0.34467924 -0.34467924 -0.34467924 ... -0.27216592 -0.27216592\n",
      "  -0.27216592]\n",
      " ...\n",
      " [-0.2638993  -0.2638993  -0.2638993  ... -0.30452207 -0.30452207\n",
      "  -0.30452207]\n",
      " [-0.30593273 -0.30593273 -0.30593273 ... -0.30109224 -0.30109224\n",
      "  -0.30109224]\n",
      " [-0.28382438 -0.28382438 -0.28382438 ... -0.28955728 -0.28955728\n",
      "  -0.28955728]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.3227744  -0.3227744  -0.3227744  ... -0.3298269  -0.3298269\n",
      "  -0.3298269 ]\n",
      " [-0.27946788 -0.27946788 -0.27946788 ... -0.2973371  -0.2973371\n",
      "  -0.2973371 ]\n",
      " [-0.3116777  -0.3116777  -0.3116777  ... -0.31368592 -0.31368592\n",
      "  -0.31368592]\n",
      " ...\n",
      " [-0.31977898 -0.31977898 -0.31977898 ... -0.28217465 -0.28217465\n",
      "  -0.28217465]\n",
      " [-0.293393   -0.293393   -0.293393   ... -0.2974066  -0.2974066\n",
      "  -0.2974066 ]\n",
      " [-0.33780164 -0.33780164 -0.33780164 ... -0.2792967  -0.2792967\n",
      "  -0.2792967 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.31916696 -0.31916696 -0.31916696 ... -0.32584828 -0.32584828\n",
      "  -0.32584828]\n",
      " [-0.305668   -0.305668   -0.305668   ... -0.317971   -0.317971\n",
      "  -0.317971  ]\n",
      " [-0.30147827 -0.30147827 -0.30147827 ... -0.32699552 -0.32699552\n",
      "  -0.32699552]\n",
      " ...\n",
      " [-0.3158459  -0.3158459  -0.3158459  ... -0.31856397 -0.31856397\n",
      "  -0.31856397]\n",
      " [-0.2945599  -0.2945599  -0.2945599  ... -0.3068571  -0.3068571\n",
      "  -0.3068571 ]\n",
      " [-0.33652583 -0.33652583 -0.33652583 ... -0.2998622  -0.2998622\n",
      "  -0.2998622 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.32488042 -0.32488042 -0.32488042 ... -0.29558662 -0.29558662\n",
      "  -0.29558662]\n",
      " [-0.3122967  -0.3122967  -0.3122967  ... -0.28648958 -0.28648958\n",
      "  -0.28648958]\n",
      " [-0.29261947 -0.29261947 -0.29261947 ... -0.30545747 -0.30545747\n",
      "  -0.30545747]\n",
      " ...\n",
      " [-0.30463007 -0.30463007 -0.30463007 ... -0.30428374 -0.30428374\n",
      "  -0.30428374]\n",
      " [-0.3097006  -0.3097006  -0.3097006  ... -0.30790204 -0.30790204\n",
      "  -0.30790204]\n",
      " [-0.3008234  -0.3008234  -0.3008234  ... -0.2927113  -0.2927113\n",
      "  -0.2927113 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.27129698 -0.27129698 -0.27129698 ... -0.3088014  -0.3088014\n",
      "  -0.3088014 ]\n",
      " [-0.30969205 -0.30969205 -0.30969205 ... -0.29849416 -0.29849416\n",
      "  -0.29849416]\n",
      " [-0.33229604 -0.33229604 -0.33229604 ... -0.25932917 -0.25932917\n",
      "  -0.25932917]\n",
      " ...\n",
      " [-0.3016154  -0.3016154  -0.3016154  ... -0.31478146 -0.31478146\n",
      "  -0.31478146]\n",
      " [-0.30324566 -0.30324566 -0.30324566 ... -0.32517788 -0.32517788\n",
      "  -0.32517788]\n",
      " [-0.27946287 -0.27946287 -0.27946287 ... -0.3072583  -0.3072583\n",
      "  -0.3072583 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2323  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.28945273 -0.28945273 -0.28945273 ... -0.27235088 -0.27235088\n",
      "  -0.27235088]\n",
      " [-0.26770192 -0.26770192 -0.26770192 ... -0.2921851  -0.2921851\n",
      "  -0.2921851 ]\n",
      " [-0.28647125 -0.28647125 -0.28647125 ... -0.2992087  -0.2992087\n",
      "  -0.2992087 ]\n",
      " ...\n",
      " [-0.31218725 -0.31218725 -0.31218725 ... -0.28494912 -0.28494912\n",
      "  -0.28494912]\n",
      " [-0.28034344 -0.28034344 -0.28034344 ... -0.2603117  -0.2603117\n",
      "  -0.2603117 ]\n",
      " [-0.29523614 -0.29523614 -0.29523614 ... -0.28905082 -0.28905082\n",
      "  -0.28905082]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.2861874  -0.2861874  -0.2861874  ... -0.2995469  -0.2995469\n",
      "  -0.2995469 ]\n",
      " [-0.30533803 -0.30533803 -0.30533803 ... -0.27235088 -0.27235088\n",
      "  -0.27235088]\n",
      " [-0.27075252 -0.27075252 -0.27075252 ... -0.29520643 -0.29520643\n",
      "  -0.29520643]\n",
      " ...\n",
      " [-0.2846182  -0.2846182  -0.2846182  ... -0.27972397 -0.27972397\n",
      "  -0.27972397]\n",
      " [-0.2925945  -0.2925945  -0.2925945  ... -0.29559815 -0.29559815\n",
      "  -0.29559815]\n",
      " [-0.3013752  -0.3013752  -0.3013752  ... -0.2973146  -0.2973146\n",
      "  -0.2973146 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.27065623 -0.27065623 -0.27065623 ... -0.2826727  -0.2826727\n",
      "  -0.2826727 ]\n",
      " [-0.29624474 -0.29624474 -0.29624474 ... -0.30956    -0.30956\n",
      "  -0.30956   ]\n",
      " [-0.27895638 -0.27895638 -0.27895638 ... -0.29882258 -0.29882258\n",
      "  -0.29882258]\n",
      " ...\n",
      " [-0.30039227 -0.30039227 -0.30039227 ... -0.3023299  -0.3023299\n",
      "  -0.3023299 ]\n",
      " [-0.31176826 -0.31176826 -0.31176826 ... -0.31209219 -0.31209219\n",
      "  -0.31209219]\n",
      " [-0.3006097  -0.3006097  -0.3006097  ... -0.27505988 -0.27505988\n",
      "  -0.27505988]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.27978528 -0.27978528 -0.27978528 ... -0.29356885 -0.29356885\n",
      "  -0.29356885]\n",
      " [-0.25644472 -0.25644472 -0.25644472 ... -0.2959255  -0.2959255\n",
      "  -0.2959255 ]\n",
      " [-0.30111682 -0.30111682 -0.30111682 ... -0.32047278 -0.32047278\n",
      "  -0.32047278]\n",
      " ...\n",
      " [-0.26566356 -0.26566356 -0.26566356 ... -0.2885173  -0.2885173\n",
      "  -0.2885173 ]\n",
      " [-0.29285604 -0.29285604 -0.29285604 ... -0.28964093 -0.28964093\n",
      "  -0.28964093]\n",
      " [-0.29983068 -0.29983068 -0.29983068 ... -0.27874798 -0.27874798\n",
      "  -0.27874798]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.30107093 -0.30107093 -0.30107093 ... -0.26137066 -0.26137066\n",
      "  -0.26137066]\n",
      " [-0.29200292 -0.29200292 -0.29200292 ... -0.26624528 -0.26624528\n",
      "  -0.26624528]\n",
      " [-0.29443175 -0.29443175 -0.29443175 ... -0.31122506 -0.31122506\n",
      "  -0.31122506]\n",
      " ...\n",
      " [-0.29971093 -0.29971093 -0.29971093 ... -0.28305954 -0.28305954\n",
      "  -0.28305954]\n",
      " [-0.31648988 -0.31648988 -0.31648988 ... -0.305343   -0.305343\n",
      "  -0.305343  ]\n",
      " [-0.29036063 -0.29036063 -0.29036063 ... -0.32307944 -0.32307944\n",
      "  -0.32307944]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.25919247 -0.25919247 -0.25919247 ... -0.30808365 -0.30808365\n",
      "  -0.30808365]\n",
      " [-0.30327135 -0.30327135 -0.30327135 ... -0.30620182 -0.30620182\n",
      "  -0.30620182]\n",
      " [-0.30088145 -0.30088145 -0.30088145 ... -0.2910318  -0.2910318\n",
      "  -0.2910318 ]\n",
      " ...\n",
      " [-0.30589512 -0.30589512 -0.30589512 ... -0.29487458 -0.29487458\n",
      "  -0.29487458]\n",
      " [-0.2909504  -0.2909504  -0.2909504  ... -0.27735645 -0.27735645\n",
      "  -0.27735645]\n",
      " [-0.29229534 -0.29229534 -0.29229534 ... -0.3000028  -0.3000028\n",
      "  -0.3000028 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28653058 -0.28653058 -0.28653058 ... -0.30556378 -0.30556378\n",
      "  -0.30556378]\n",
      " [-0.27854708 -0.27854708 -0.27854708 ... -0.28895676 -0.28895676\n",
      "  -0.28895676]\n",
      " [-0.3073141  -0.3073141  -0.3073141  ... -0.31710806 -0.31710806\n",
      "  -0.31710806]\n",
      " ...\n",
      " [-0.28779125 -0.28779125 -0.28779125 ... -0.28871554 -0.28871554\n",
      "  -0.28871554]\n",
      " [-0.28569937 -0.28569937 -0.28569937 ... -0.3266698  -0.3266698\n",
      "  -0.3266698 ]\n",
      " [-0.28545845 -0.28545845 -0.28545845 ... -0.2804706  -0.2804706\n",
      "  -0.2804706 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.3064075  -0.3064075  -0.3064075  ... -0.28608635 -0.28608635\n",
      "  -0.28608635]\n",
      " [-0.29404697 -0.29404697 -0.29404697 ... -0.30994365 -0.30994365\n",
      "  -0.30994365]\n",
      " [-0.25887042 -0.25887042 -0.25887042 ... -0.2890703  -0.2890703\n",
      "  -0.2890703 ]\n",
      " ...\n",
      " [-0.2752221  -0.2752221  -0.2752221  ... -0.3057606  -0.3057606\n",
      "  -0.3057606 ]\n",
      " [-0.32997125 -0.32997125 -0.32997125 ... -0.29455894 -0.29455894\n",
      "  -0.29455894]\n",
      " [-0.29111165 -0.29111165 -0.29111165 ... -0.26487505 -0.26487505\n",
      "  -0.26487505]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2424  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2567131  -0.2567131  -0.2567131  ... -0.30278057 -0.30278057\n",
      "  -0.30278057]\n",
      " [-0.2775476  -0.2775476  -0.2775476  ... -0.30012667 -0.30012667\n",
      "  -0.30012667]\n",
      " [-0.25717944 -0.25717944 -0.25717944 ... -0.27219215 -0.27219215\n",
      "  -0.27219215]\n",
      " ...\n",
      " [-0.2821014  -0.2821014  -0.2821014  ... -0.30334422 -0.30334422\n",
      "  -0.30334422]\n",
      " [-0.29284927 -0.29284927 -0.29284927 ... -0.2849235  -0.2849235\n",
      "  -0.2849235 ]\n",
      " [-0.28476146 -0.28476146 -0.28476146 ... -0.28678855 -0.28678855\n",
      "  -0.28678855]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.26302412 -0.26302412 -0.26302412 ... -0.27889663 -0.27889663\n",
      "  -0.27889663]\n",
      " [-0.28663072 -0.28663072 -0.28663072 ... -0.28841126 -0.28841126\n",
      "  -0.28841126]\n",
      " [-0.29362434 -0.29362434 -0.29362434 ... -0.2565841  -0.2565841\n",
      "  -0.2565841 ]\n",
      " ...\n",
      " [-0.2906657  -0.2906657  -0.2906657  ... -0.27355194 -0.27355194\n",
      "  -0.27355194]\n",
      " [-0.2710899  -0.2710899  -0.2710899  ... -0.2678509  -0.2678509\n",
      "  -0.2678509 ]\n",
      " [-0.2796637  -0.2796637  -0.2796637  ... -0.28960466 -0.28960466\n",
      "  -0.28960466]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2697147  -0.2697147  -0.2697147  ... -0.26123008 -0.26123008\n",
      "  -0.26123008]\n",
      " [-0.28207365 -0.28207365 -0.28207365 ... -0.27065632 -0.27065632\n",
      "  -0.27065632]\n",
      " [-0.29819393 -0.29819393 -0.29819393 ... -0.28863835 -0.28863835\n",
      "  -0.28863835]\n",
      " ...\n",
      " [-0.25846922 -0.25846922 -0.25846922 ... -0.2926087  -0.2926087\n",
      "  -0.2926087 ]\n",
      " [-0.2795515  -0.2795515  -0.2795515  ... -0.28411654 -0.28411654\n",
      "  -0.28411654]\n",
      " [-0.24914856 -0.24914856 -0.24914856 ... -0.2854468  -0.2854468\n",
      "  -0.2854468 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24936199 -0.24936199 -0.24936199 ... -0.26393253 -0.26393253\n",
      "  -0.26393253]\n",
      " [-0.27850622 -0.27850622 -0.27850622 ... -0.29159224 -0.29159224\n",
      "  -0.29159224]\n",
      " [-0.2596076  -0.2596076  -0.2596076  ... -0.28023306 -0.28023306\n",
      "  -0.28023306]\n",
      " ...\n",
      " [-0.2540068  -0.2540068  -0.2540068  ... -0.2899875  -0.2899875\n",
      "  -0.2899875 ]\n",
      " [-0.26052466 -0.26052466 -0.26052466 ... -0.28449118 -0.28449118\n",
      "  -0.28449118]\n",
      " [-0.26792842 -0.26792842 -0.26792842 ... -0.29072338 -0.29072338\n",
      "  -0.29072338]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2783395  -0.2783395  -0.2783395  ... -0.26254717 -0.26254717\n",
      "  -0.26254717]\n",
      " [-0.28537008 -0.28537008 -0.28537008 ... -0.29203886 -0.29203886\n",
      "  -0.29203886]\n",
      " [-0.30213028 -0.30213028 -0.30213028 ... -0.30334422 -0.30334422\n",
      "  -0.30334422]\n",
      " ...\n",
      " [-0.29492742 -0.29492742 -0.29492742 ... -0.30114916 -0.30114916\n",
      "  -0.30114916]\n",
      " [-0.28009176 -0.28009176 -0.28009176 ... -0.26556987 -0.26556987\n",
      "  -0.26556987]\n",
      " [-0.3019367  -0.3019367  -0.3019367  ... -0.27844056 -0.27844056\n",
      "  -0.27844056]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.28777435 -0.28777435 -0.28777435 ... -0.26763028 -0.26763028\n",
      "  -0.26763028]\n",
      " [-0.2730748  -0.2730748  -0.2730748  ... -0.28799504 -0.28799504\n",
      "  -0.28799504]\n",
      " [-0.28461662 -0.28461662 -0.28461662 ... -0.257796   -0.257796\n",
      "  -0.257796  ]\n",
      " ...\n",
      " [-0.27208477 -0.27208477 -0.27208477 ... -0.28623772 -0.28623772\n",
      "  -0.28623772]\n",
      " [-0.28056982 -0.28056982 -0.28056982 ... -0.30582836 -0.30582836\n",
      "  -0.30582836]\n",
      " [-0.30676466 -0.30676466 -0.30676466 ... -0.25265175 -0.25265175\n",
      "  -0.25265175]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.27835205 -0.27835205 -0.27835205 ... -0.26846975 -0.26846975\n",
      "  -0.26846975]\n",
      " [-0.2727582  -0.2727582  -0.2727582  ... -0.26903158 -0.26903158\n",
      "  -0.26903158]\n",
      " [-0.28719553 -0.28719553 -0.28719553 ... -0.2757406  -0.2757406\n",
      "  -0.2757406 ]\n",
      " ...\n",
      " [-0.26629984 -0.26629984 -0.26629984 ... -0.27138224 -0.27138224\n",
      "  -0.27138224]\n",
      " [-0.25504905 -0.25504905 -0.25504905 ... -0.2813045  -0.2813045\n",
      "  -0.2813045 ]\n",
      " [-0.23286803 -0.23286803 -0.23286803 ... -0.26034564 -0.26034564\n",
      "  -0.26034564]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.27742672 -0.27742672 -0.27742672 ... -0.27477282 -0.27477282\n",
      "  -0.27477282]\n",
      " [-0.29696137 -0.29696137 -0.29696137 ... -0.28446564 -0.28446564\n",
      "  -0.28446564]\n",
      " [-0.3059402  -0.3059402  -0.3059402  ... -0.28060108 -0.28060108\n",
      "  -0.28060108]\n",
      " ...\n",
      " [-0.2766541  -0.2766541  -0.2766541  ... -0.2622071  -0.2622071\n",
      "  -0.2622071 ]\n",
      " [-0.25969976 -0.25969976 -0.25969976 ... -0.27893505 -0.27893505\n",
      "  -0.27893505]\n",
      " [-0.2498092  -0.2498092  -0.2498092  ... -0.25671145 -0.25671145\n",
      "  -0.25671145]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2525  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2622416  -0.2622416  -0.2622416  ... -0.25123233 -0.25123233\n",
      "  -0.25123233]\n",
      " [-0.26201206 -0.26201206 -0.26201206 ... -0.2591816  -0.2591816\n",
      "  -0.2591816 ]\n",
      " [-0.25629735 -0.25629735 -0.25629735 ... -0.26861274 -0.26861274\n",
      "  -0.26861274]\n",
      " ...\n",
      " [-0.23673081 -0.23673081 -0.23673081 ... -0.24733135 -0.24733135\n",
      "  -0.24733135]\n",
      " [-0.2821567  -0.2821567  -0.2821567  ... -0.23943853 -0.23943853\n",
      "  -0.23943853]\n",
      " [-0.28522238 -0.28522238 -0.28522238 ... -0.23966223 -0.23966223\n",
      "  -0.23966223]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.27378708 -0.27378708 -0.27378708 ... -0.22966883 -0.22966883\n",
      "  -0.22966883]\n",
      " [-0.2612232  -0.2612232  -0.2612232  ... -0.25594586 -0.25594586\n",
      "  -0.25594586]\n",
      " [-0.2613172  -0.2613172  -0.2613172  ... -0.26347286 -0.26347286\n",
      "  -0.26347286]\n",
      " ...\n",
      " [-0.24722488 -0.24722488 -0.24722488 ... -0.2528022  -0.2528022\n",
      "  -0.2528022 ]\n",
      " [-0.23058142 -0.23058142 -0.23058142 ... -0.26912624 -0.26912624\n",
      "  -0.26912624]\n",
      " [-0.26019418 -0.26019418 -0.26019418 ... -0.27772707 -0.27772707\n",
      "  -0.27772707]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24654219 -0.24654219 -0.24654219 ... -0.2721447  -0.2721447\n",
      "  -0.2721447 ]\n",
      " [-0.26810545 -0.26810545 -0.26810545 ... -0.22172897 -0.22172897\n",
      "  -0.22172897]\n",
      " [-0.2691009  -0.2691009  -0.2691009  ... -0.2652687  -0.2652687\n",
      "  -0.2652687 ]\n",
      " ...\n",
      " [-0.24479596 -0.24479596 -0.24479596 ... -0.25757056 -0.25757056\n",
      "  -0.25757056]\n",
      " [-0.27287993 -0.27287993 -0.27287993 ... -0.2730867  -0.2730867\n",
      "  -0.2730867 ]\n",
      " [-0.30745733 -0.30745733 -0.30745733 ... -0.25877377 -0.25877377\n",
      "  -0.25877377]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.21884255 -0.21884255 -0.21884255 ... -0.23755163 -0.23755163\n",
      "  -0.23755163]\n",
      " [-0.25151432 -0.25151432 -0.25151432 ... -0.26818442 -0.26818442\n",
      "  -0.26818442]\n",
      " [-0.25937244 -0.25937244 -0.25937244 ... -0.2492171  -0.2492171\n",
      "  -0.2492171 ]\n",
      " ...\n",
      " [-0.257313   -0.257313   -0.257313   ... -0.2553874  -0.2553874\n",
      "  -0.2553874 ]\n",
      " [-0.2661806  -0.2661806  -0.2661806  ... -0.25263476 -0.25263476\n",
      "  -0.25263476]\n",
      " [-0.2461313  -0.2461313  -0.2461313  ... -0.23047023 -0.23047023\n",
      "  -0.23047023]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23426118 -0.23426118 -0.23426118 ... -0.2212118  -0.2212118\n",
      "  -0.2212118 ]\n",
      " [-0.2599293  -0.2599293  -0.2599293  ... -0.26001242 -0.26001242\n",
      "  -0.26001242]\n",
      " [-0.25414664 -0.25414664 -0.25414664 ... -0.24653532 -0.24653532\n",
      "  -0.24653532]\n",
      " ...\n",
      " [-0.27119908 -0.27119908 -0.27119908 ... -0.273623   -0.273623\n",
      "  -0.273623  ]\n",
      " [-0.26986867 -0.26986867 -0.26986867 ... -0.25361493 -0.25361493\n",
      "  -0.25361493]\n",
      " [-0.25177243 -0.25177243 -0.25177243 ... -0.25761372 -0.25761372\n",
      "  -0.25761372]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.28295678 -0.28295678 -0.28295678 ... -0.23817316 -0.23817316\n",
      "  -0.23817316]\n",
      " [-0.25583434 -0.25583434 -0.25583434 ... -0.26974192 -0.26974192\n",
      "  -0.26974192]\n",
      " [-0.2742515  -0.2742515  -0.2742515  ... -0.2644067  -0.2644067\n",
      "  -0.2644067 ]\n",
      " ...\n",
      " [-0.25559163 -0.25559163 -0.25559163 ... -0.2700679  -0.2700679\n",
      "  -0.2700679 ]\n",
      " [-0.29512548 -0.29512548 -0.29512548 ... -0.27191186 -0.27191186\n",
      "  -0.27191186]\n",
      " [-0.2373337  -0.2373337  -0.2373337  ... -0.25134245 -0.25134245\n",
      "  -0.25134245]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24414527 -0.24414527 -0.24414527 ... -0.252931   -0.252931\n",
      "  -0.252931  ]\n",
      " [-0.2703026  -0.2703026  -0.2703026  ... -0.2621729  -0.2621729\n",
      "  -0.2621729 ]\n",
      " [-0.26031947 -0.26031947 -0.26031947 ... -0.24094236 -0.24094236\n",
      "  -0.24094236]\n",
      " ...\n",
      " [-0.28558725 -0.28558725 -0.28558725 ... -0.28493983 -0.28493983\n",
      "  -0.28493983]\n",
      " [-0.24641894 -0.24641894 -0.24641894 ... -0.26546502 -0.26546502\n",
      "  -0.26546502]\n",
      " [-0.251615   -0.251615   -0.251615   ... -0.26407164 -0.26407164\n",
      "  -0.26407164]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.23725924 -0.23725924 -0.23725924 ... -0.25266805 -0.25266805\n",
      "  -0.25266805]\n",
      " [-0.2494056  -0.2494056  -0.2494056  ... -0.26149118 -0.26149118\n",
      "  -0.26149118]\n",
      " [-0.2720555  -0.2720555  -0.2720555  ... -0.2543949  -0.2543949\n",
      "  -0.2543949 ]\n",
      " ...\n",
      " [-0.29025048 -0.29025048 -0.29025048 ... -0.24260736 -0.24260736\n",
      "  -0.24260736]\n",
      " [-0.2781966  -0.2781966  -0.2781966  ... -0.30659485 -0.30659485\n",
      "  -0.30659485]\n",
      " [-0.2237446  -0.2237446  -0.2237446  ... -0.2621758  -0.2621758\n",
      "  -0.2621758 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2626  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.22448099 -0.22448099 -0.22448099 ... -0.21163027 -0.21163027\n",
      "  -0.21163027]\n",
      " [-0.23625325 -0.23625325 -0.23625325 ... -0.24281013 -0.24281013\n",
      "  -0.24281013]\n",
      " [-0.23374061 -0.23374061 -0.23374061 ... -0.22528075 -0.22528075\n",
      "  -0.22528075]\n",
      " ...\n",
      " [-0.23152034 -0.23152034 -0.23152034 ... -0.23555991 -0.23555991\n",
      "  -0.23555991]\n",
      " [-0.23009475 -0.23009475 -0.23009475 ... -0.23660903 -0.23660903\n",
      "  -0.23660903]\n",
      " [-0.23614287 -0.23614287 -0.23614287 ... -0.22945824 -0.22945824\n",
      "  -0.22945824]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22541651 -0.22541651 -0.22541651 ... -0.22469264 -0.22469264\n",
      "  -0.22469264]\n",
      " [-0.24331926 -0.24331926 -0.24331926 ... -0.22683157 -0.22683157\n",
      "  -0.22683157]\n",
      " [-0.22427785 -0.22427785 -0.22427785 ... -0.24473147 -0.24473147\n",
      "  -0.24473147]\n",
      " ...\n",
      " [-0.24483548 -0.24483548 -0.24483548 ... -0.21954143 -0.21954143\n",
      "  -0.21954143]\n",
      " [-0.22903821 -0.22903821 -0.22903821 ... -0.2502366  -0.2502366\n",
      "  -0.2502366 ]\n",
      " [-0.24904177 -0.24904177 -0.24904177 ... -0.22547626 -0.22547626\n",
      "  -0.22547626]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.19372171 -0.19372171 -0.19372171 ... -0.22786888 -0.22786888\n",
      "  -0.22786888]\n",
      " [-0.22531645 -0.22531645 -0.22531645 ... -0.27849892 -0.27849892\n",
      "  -0.27849892]\n",
      " [-0.22565427 -0.22565427 -0.22565427 ... -0.24682489 -0.24682489\n",
      "  -0.24682489]\n",
      " ...\n",
      " [-0.23327011 -0.23327011 -0.23327011 ... -0.22729336 -0.22729336\n",
      "  -0.22729336]\n",
      " [-0.24820817 -0.24820817 -0.24820817 ... -0.22985105 -0.22985105\n",
      "  -0.22985105]\n",
      " [-0.23610663 -0.23610663 -0.23610663 ... -0.24745488 -0.24745488\n",
      "  -0.24745488]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.23808125 -0.23808125 -0.23808125 ... -0.23604749 -0.23604749\n",
      "  -0.23604749]\n",
      " [-0.23178303 -0.23178303 -0.23178303 ... -0.23040263 -0.23040263\n",
      "  -0.23040263]\n",
      " [-0.22349074 -0.22349074 -0.22349074 ... -0.2321468  -0.2321468\n",
      "  -0.2321468 ]\n",
      " ...\n",
      " [-0.27942908 -0.27942908 -0.27942908 ... -0.2556815  -0.2556815\n",
      "  -0.2556815 ]\n",
      " [-0.22043434 -0.22043434 -0.22043434 ... -0.25438082 -0.25438082\n",
      "  -0.25438082]\n",
      " [-0.23790854 -0.23790854 -0.23790854 ... -0.22830303 -0.22830303\n",
      "  -0.22830303]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20331605 -0.20331605 -0.20331605 ... -0.24649958 -0.24649958\n",
      "  -0.24649958]\n",
      " [-0.2649289  -0.2649289  -0.2649289  ... -0.2227771  -0.2227771\n",
      "  -0.2227771 ]\n",
      " [-0.23291008 -0.23291008 -0.23291008 ... -0.27009806 -0.27009806\n",
      "  -0.27009806]\n",
      " ...\n",
      " [-0.23927948 -0.23927948 -0.23927948 ... -0.24216457 -0.24216457\n",
      "  -0.24216457]\n",
      " [-0.22370991 -0.22370991 -0.22370991 ... -0.2237774  -0.2237774\n",
      "  -0.2237774 ]\n",
      " [-0.22651038 -0.22651038 -0.22651038 ... -0.21587408 -0.21587408\n",
      "  -0.21587408]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.25728786 -0.25728786 -0.25728786 ... -0.26064253 -0.26064253\n",
      "  -0.26064253]\n",
      " [-0.24723393 -0.24723393 -0.24723393 ... -0.21807657 -0.21807657\n",
      "  -0.21807657]\n",
      " [-0.25506443 -0.25506443 -0.25506443 ... -0.24831721 -0.24831721\n",
      "  -0.24831721]\n",
      " ...\n",
      " [-0.27103567 -0.27103567 -0.27103567 ... -0.22737162 -0.22737162\n",
      "  -0.22737162]\n",
      " [-0.22401662 -0.22401662 -0.22401662 ... -0.24344382 -0.24344382\n",
      "  -0.24344382]\n",
      " [-0.25443858 -0.25443858 -0.25443858 ... -0.23631513 -0.23631513\n",
      "  -0.23631513]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23435353 -0.23435353 -0.23435353 ... -0.23035222 -0.23035222\n",
      "  -0.23035222]\n",
      " [-0.22183904 -0.22183904 -0.22183904 ... -0.2435421  -0.2435421\n",
      "  -0.2435421 ]\n",
      " [-0.25152817 -0.25152817 -0.25152817 ... -0.25714487 -0.25714487\n",
      "  -0.25714487]\n",
      " ...\n",
      " [-0.22655539 -0.22655539 -0.22655539 ... -0.22280881 -0.22280881\n",
      "  -0.22280881]\n",
      " [-0.2350487  -0.2350487  -0.2350487  ... -0.2297213  -0.2297213\n",
      "  -0.2297213 ]\n",
      " [-0.26909995 -0.26909995 -0.26909995 ... -0.25147903 -0.25147903\n",
      "  -0.25147903]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24004287 -0.24004287 -0.24004287 ... -0.2331863  -0.2331863\n",
      "  -0.2331863 ]\n",
      " [-0.23995253 -0.23995253 -0.23995253 ... -0.22826788 -0.22826788\n",
      "  -0.22826788]\n",
      " [-0.221668   -0.221668   -0.221668   ... -0.24355245 -0.24355245\n",
      "  -0.24355245]\n",
      " ...\n",
      " [-0.2135517  -0.2135517  -0.2135517  ... -0.23000392 -0.23000392\n",
      "  -0.23000392]\n",
      " [-0.24062595 -0.24062595 -0.24062595 ... -0.2520074  -0.2520074\n",
      "  -0.2520074 ]\n",
      " [-0.22703864 -0.22703864 -0.22703864 ... -0.2199077  -0.2199077\n",
      "  -0.2199077 ]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2727  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23473534 -0.23473534 -0.23473534 ... -0.21780834 -0.21780834\n",
      "  -0.21780834]\n",
      " [-0.17271009 -0.17271009 -0.17271009 ... -0.22644202 -0.22644202\n",
      "  -0.22644202]\n",
      " [-0.19797114 -0.19797114 -0.19797114 ... -0.21184528 -0.21184528\n",
      "  -0.21184528]\n",
      " ...\n",
      " [-0.2295067  -0.2295067  -0.2295067  ... -0.1999405  -0.1999405\n",
      "  -0.1999405 ]\n",
      " [-0.22135058 -0.22135058 -0.22135058 ... -0.22929412 -0.22929412\n",
      "  -0.22929412]\n",
      " [-0.20770936 -0.20770936 -0.20770936 ... -0.21826234 -0.21826234\n",
      "  -0.21826234]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22757155 -0.22757155 -0.22757155 ... -0.22874677 -0.22874677\n",
      "  -0.22874677]\n",
      " [-0.19820105 -0.19820105 -0.19820105 ... -0.21404204 -0.21404204\n",
      "  -0.21404204]\n",
      " [-0.19049624 -0.19049624 -0.19049624 ... -0.18361859 -0.18361859\n",
      "  -0.18361859]\n",
      " ...\n",
      " [-0.22357434 -0.22357434 -0.22357434 ... -0.19714376 -0.19714376\n",
      "  -0.19714376]\n",
      " [-0.2135785  -0.2135785  -0.2135785  ... -0.20987692 -0.20987692\n",
      "  -0.20987692]\n",
      " [-0.23261261 -0.23261261 -0.23261261 ... -0.21303417 -0.21303417\n",
      "  -0.21303417]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21553358 -0.21553358 -0.21553358 ... -0.21645094 -0.21645094\n",
      "  -0.21645094]\n",
      " [-0.19303274 -0.19303274 -0.19303274 ... -0.20255814 -0.20255814\n",
      "  -0.20255814]\n",
      " [-0.22345293 -0.22345293 -0.22345293 ... -0.25280356 -0.25280356\n",
      "  -0.25280356]\n",
      " ...\n",
      " [-0.21447252 -0.21447252 -0.21447252 ... -0.23649165 -0.23649165\n",
      "  -0.23649165]\n",
      " [-0.23276687 -0.23276687 -0.23276687 ... -0.22232565 -0.22232565\n",
      "  -0.22232565]\n",
      " [-0.23298874 -0.23298874 -0.23298874 ... -0.2133041  -0.2133041\n",
      "  -0.2133041 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.20967019 -0.20967019 -0.20967019 ... -0.2317757  -0.2317757\n",
      "  -0.2317757 ]\n",
      " [-0.22892408 -0.22892408 -0.22892408 ... -0.20029801 -0.20029801\n",
      "  -0.20029801]\n",
      " [-0.2095201  -0.2095201  -0.2095201  ... -0.2519606  -0.2519606\n",
      "  -0.2519606 ]\n",
      " ...\n",
      " [-0.2107463  -0.2107463  -0.2107463  ... -0.21303417 -0.21303417\n",
      "  -0.21303417]\n",
      " [-0.23247884 -0.23247884 -0.23247884 ... -0.22775011 -0.22775011\n",
      "  -0.22775011]\n",
      " [-0.22323306 -0.22323306 -0.22323306 ... -0.22112201 -0.22112201\n",
      "  -0.22112201]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20522068 -0.20522068 -0.20522068 ... -0.17386407 -0.17386407\n",
      "  -0.17386407]\n",
      " [-0.20582338 -0.20582338 -0.20582338 ... -0.1724616  -0.1724616\n",
      "  -0.1724616 ]\n",
      " [-0.21883851 -0.21883851 -0.21883851 ... -0.21649101 -0.21649101\n",
      "  -0.21649101]\n",
      " ...\n",
      " [-0.21322626 -0.21322626 -0.21322626 ... -0.20864293 -0.20864293\n",
      "  -0.20864293]\n",
      " [-0.19456324 -0.19456324 -0.19456324 ... -0.23707949 -0.23707949\n",
      "  -0.23707949]\n",
      " [-0.2080102  -0.2080102  -0.2080102  ... -0.22417824 -0.22417824\n",
      "  -0.22417824]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19902992 -0.19902992 -0.19902992 ... -0.22370341 -0.22370341\n",
      "  -0.22370341]\n",
      " [-0.18463802 -0.18463802 -0.18463802 ... -0.1998148  -0.1998148\n",
      "  -0.1998148 ]\n",
      " [-0.22839811 -0.22839811 -0.22839811 ... -0.20203231 -0.20203231\n",
      "  -0.20203231]\n",
      " ...\n",
      " [-0.21657936 -0.21657936 -0.21657936 ... -0.22469077 -0.22469077\n",
      "  -0.22469077]\n",
      " [-0.20899688 -0.20899688 -0.20899688 ... -0.19928072 -0.19928072\n",
      "  -0.19928072]\n",
      " [-0.21607625 -0.21607625 -0.21607625 ... -0.22923589 -0.22923589\n",
      "  -0.22923589]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23284985 -0.23284985 -0.23284985 ... -0.24450321 -0.24450321\n",
      "  -0.24450321]\n",
      " [-0.22656447 -0.22656447 -0.22656447 ... -0.21506225 -0.21506225\n",
      "  -0.21506225]\n",
      " [-0.21171102 -0.21171102 -0.21171102 ... -0.2321813  -0.2321813\n",
      "  -0.2321813 ]\n",
      " ...\n",
      " [-0.20330903 -0.20330903 -0.20330903 ... -0.22860697 -0.22860697\n",
      "  -0.22860697]\n",
      " [-0.24369773 -0.24369773 -0.24369773 ... -0.22220233 -0.22220233\n",
      "  -0.22220233]\n",
      " [-0.22286782 -0.22286782 -0.22286782 ... -0.20902434 -0.20902434\n",
      "  -0.20902434]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18956324 -0.18956324 -0.18956324 ... -0.16654909 -0.16654909\n",
      "  -0.16654909]\n",
      " [-0.21718231 -0.21718231 -0.21718231 ... -0.19668604 -0.19668604\n",
      "  -0.19668604]\n",
      " [-0.21985346 -0.21985346 -0.21985346 ... -0.20692104 -0.20692104\n",
      "  -0.20692104]\n",
      " ...\n",
      " [-0.21965955 -0.21965955 -0.21965955 ... -0.23420899 -0.23420899\n",
      "  -0.23420899]\n",
      " [-0.21000136 -0.21000136 -0.21000136 ... -0.23009168 -0.23009168\n",
      "  -0.23009168]\n",
      " [-0.21479179 -0.21479179 -0.21479179 ... -0.21427312 -0.21427312\n",
      "  -0.21427312]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2828  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.18491055 -0.18491055 -0.18491055 ... -0.19221902 -0.19221902\n",
      "  -0.19221902]\n",
      " [-0.20889646 -0.20889646 -0.20889646 ... -0.18805772 -0.18805772\n",
      "  -0.18805772]\n",
      " [-0.20084135 -0.20084135 -0.20084135 ... -0.17937809 -0.17937809\n",
      "  -0.17937809]\n",
      " ...\n",
      " [-0.19128577 -0.19128577 -0.19128577 ... -0.21834877 -0.21834877\n",
      "  -0.21834877]\n",
      " [-0.21207005 -0.21207005 -0.21207005 ... -0.20009594 -0.20009594\n",
      "  -0.20009594]\n",
      " [-0.1650943  -0.1650943  -0.1650943  ... -0.17464852 -0.17464852\n",
      "  -0.17464852]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19547759 -0.19547759 -0.19547759 ... -0.19719884 -0.19719884\n",
      "  -0.19719884]\n",
      " [-0.19039047 -0.19039047 -0.19039047 ... -0.21183692 -0.21183692\n",
      "  -0.21183692]\n",
      " [-0.17185518 -0.17185518 -0.17185518 ... -0.19030416 -0.19030416\n",
      "  -0.19030416]\n",
      " ...\n",
      " [-0.20145334 -0.20145334 -0.20145334 ... -0.19280586 -0.19280586\n",
      "  -0.19280586]\n",
      " [-0.19623403 -0.19623403 -0.19623403 ... -0.22001034 -0.22001034\n",
      "  -0.22001034]\n",
      " [-0.204828   -0.204828   -0.204828   ... -0.17170863 -0.17170863\n",
      "  -0.17170863]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15805362 -0.15805362 -0.15805362 ... -0.2058102  -0.2058102\n",
      "  -0.2058102 ]\n",
      " [-0.22473264 -0.22473264 -0.22473264 ... -0.2061558  -0.2061558\n",
      "  -0.2061558 ]\n",
      " [-0.20049646 -0.20049646 -0.20049646 ... -0.19583948 -0.19583948\n",
      "  -0.19583948]\n",
      " ...\n",
      " [-0.19570157 -0.19570157 -0.19570157 ... -0.21252513 -0.21252513\n",
      "  -0.21252513]\n",
      " [-0.18359551 -0.18359551 -0.18359551 ... -0.17810354 -0.17810354\n",
      "  -0.17810354]\n",
      " [-0.19239768 -0.19239768 -0.19239768 ... -0.16912332 -0.16912332\n",
      "  -0.16912332]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18219107 -0.18219107 -0.18219107 ... -0.2008859  -0.2008859\n",
      "  -0.2008859 ]\n",
      " [-0.21359813 -0.21359813 -0.21359813 ... -0.18181114 -0.18181114\n",
      "  -0.18181114]\n",
      " [-0.18754439 -0.18754439 -0.18754439 ... -0.18818471 -0.18818471\n",
      "  -0.18818471]\n",
      " ...\n",
      " [-0.2208531  -0.2208531  -0.2208531  ... -0.20257518 -0.20257518\n",
      "  -0.20257518]\n",
      " [-0.18808484 -0.18808484 -0.18808484 ... -0.22004062 -0.22004062\n",
      "  -0.22004062]\n",
      " [-0.20833148 -0.20833148 -0.20833148 ... -0.22156791 -0.22156791\n",
      "  -0.22156791]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.15625171 -0.15625171 -0.15625171 ... -0.22421753 -0.22421753\n",
      "  -0.22421753]\n",
      " [-0.18535993 -0.18535993 -0.18535993 ... -0.19527435 -0.19527435\n",
      "  -0.19527435]\n",
      " [-0.16356798 -0.16356798 -0.16356798 ... -0.20920229 -0.20920229\n",
      "  -0.20920229]\n",
      " ...\n",
      " [-0.15948589 -0.15948589 -0.15948589 ... -0.19727905 -0.19727905\n",
      "  -0.19727905]\n",
      " [-0.20972228 -0.20972228 -0.20972228 ... -0.2057939  -0.2057939\n",
      "  -0.2057939 ]\n",
      " [-0.22022368 -0.22022368 -0.22022368 ... -0.19431953 -0.19431953\n",
      "  -0.19431953]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19160831 -0.19160831 -0.19160831 ... -0.18935217 -0.18935217\n",
      "  -0.18935217]\n",
      " [-0.19834432 -0.19834432 -0.19834432 ... -0.17992145 -0.17992145\n",
      "  -0.17992145]\n",
      " [-0.16641375 -0.16641375 -0.16641375 ... -0.18862331 -0.18862331\n",
      "  -0.18862331]\n",
      " ...\n",
      " [-0.20468545 -0.20468545 -0.20468545 ... -0.18401584 -0.18401584\n",
      "  -0.18401584]\n",
      " [-0.18739638 -0.18739638 -0.18739638 ... -0.19530898 -0.19530898\n",
      "  -0.19530898]\n",
      " [-0.21415822 -0.21415822 -0.21415822 ... -0.22896719 -0.22896719\n",
      "  -0.22896719]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.19687204 -0.19687204 -0.19687204 ... -0.19864075 -0.19864075\n",
      "  -0.19864075]\n",
      " [-0.21311033 -0.21311033 -0.21311033 ... -0.19015823 -0.19015823\n",
      "  -0.19015823]\n",
      " [-0.20467316 -0.20467316 -0.20467316 ... -0.19640802 -0.19640802\n",
      "  -0.19640802]\n",
      " ...\n",
      " [-0.18505304 -0.18505304 -0.18505304 ... -0.18938407 -0.18938407\n",
      "  -0.18938407]\n",
      " [-0.19804095 -0.19804095 -0.19804095 ... -0.20969479 -0.20969479\n",
      "  -0.20969479]\n",
      " [-0.19299953 -0.19299953 -0.19299953 ... -0.21378922 -0.21378922\n",
      "  -0.21378922]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.20102842 -0.20102842 -0.20102842 ... -0.19069362 -0.19069362\n",
      "  -0.19069362]\n",
      " [-0.21758354 -0.21758354 -0.21758354 ... -0.20436001 -0.20436001\n",
      "  -0.20436001]\n",
      " [-0.2365523  -0.2365523  -0.2365523  ... -0.20428719 -0.20428719\n",
      "  -0.20428719]\n",
      " ...\n",
      " [-0.22102119 -0.22102119 -0.22102119 ... -0.20535803 -0.20535803\n",
      "  -0.20535803]\n",
      " [-0.19060534 -0.19060534 -0.19060534 ... -0.19526431 -0.19526431\n",
      "  -0.19526431]\n",
      " [-0.06901471 -0.06901471 -0.06901471 ... -0.21138522 -0.21138522\n",
      "  -0.21138522]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   2929  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.19452436 -0.19452436 -0.19452436 ... -0.16876271 -0.16876271\n",
      "  -0.16876271]\n",
      " [-0.19309554 -0.19309554 -0.19309554 ... -0.1989238  -0.1989238\n",
      "  -0.1989238 ]\n",
      " [-0.15377468 -0.15377468 -0.15377468 ... -0.18585157 -0.18585157\n",
      "  -0.18585157]\n",
      " ...\n",
      " [-0.13628823 -0.13628823 -0.13628823 ... -0.18347248 -0.18347248\n",
      "  -0.18347248]\n",
      " [-0.1841738  -0.1841738  -0.1841738  ... -0.15601224 -0.15601224\n",
      "  -0.15601224]\n",
      " [-0.17131811 -0.17131811 -0.17131811 ... -0.18966295 -0.18966295\n",
      "  -0.18966295]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13662231 -0.13662231 -0.13662231 ... -0.17550758 -0.17550758\n",
      "  -0.17550758]\n",
      " [-0.17633325 -0.17633325 -0.17633325 ... -0.17632282 -0.17632282\n",
      "  -0.17632282]\n",
      " [-0.16222611 -0.16222611 -0.16222611 ... -0.16365233 -0.16365233\n",
      "  -0.16365233]\n",
      " ...\n",
      " [-0.14098318 -0.14098318 -0.14098318 ... -0.17544554 -0.17544554\n",
      "  -0.17544554]\n",
      " [-0.17798547 -0.17798547 -0.17798547 ... -0.1691412  -0.1691412\n",
      "  -0.1691412 ]\n",
      " [-0.19359651 -0.19359651 -0.19359651 ... -0.16471869 -0.16471869\n",
      "  -0.16471869]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1640982  -0.1640982  -0.1640982  ... -0.18255432 -0.18255432\n",
      "  -0.18255432]\n",
      " [-0.16371134 -0.16371134 -0.16371134 ... -0.18650596 -0.18650596\n",
      "  -0.18650596]\n",
      " [-0.1923106  -0.1923106  -0.1923106  ... -0.17679797 -0.17679797\n",
      "  -0.17679797]\n",
      " ...\n",
      " [-0.16982576 -0.16982576 -0.16982576 ... -0.18715015 -0.18715015\n",
      "  -0.18715015]\n",
      " [-0.17889172 -0.17889172 -0.17889172 ... -0.16479453 -0.16479453\n",
      "  -0.16479453]\n",
      " [-0.18427688 -0.18427688 -0.18427688 ... -0.18951933 -0.18951933\n",
      "  -0.18951933]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.20179331 -0.20179331 -0.20179331 ... -0.18756361 -0.18756361\n",
      "  -0.18756361]\n",
      " [-0.1502113  -0.1502113  -0.1502113  ... -0.17722616 -0.17722616\n",
      "  -0.17722616]\n",
      " [-0.18006954 -0.18006954 -0.18006954 ... -0.1906654  -0.1906654\n",
      "  -0.1906654 ]\n",
      " ...\n",
      " [-0.19659318 -0.19659318 -0.19659318 ... -0.18282065 -0.18282065\n",
      "  -0.18282065]\n",
      " [-0.2044534  -0.2044534  -0.2044534  ... -0.18354452 -0.18354452\n",
      "  -0.18354452]\n",
      " [-0.20083722 -0.20083722 -0.20083722 ... -0.18043436 -0.18043436\n",
      "  -0.18043436]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16555932 -0.16555932 -0.16555932 ... -0.17606448 -0.17606448\n",
      "  -0.17606448]\n",
      " [-0.17794503 -0.17794503 -0.17794503 ... -0.16321912 -0.16321912\n",
      "  -0.16321912]\n",
      " [-0.16010511 -0.16010511 -0.16010511 ... -0.18203674 -0.18203674\n",
      "  -0.18203674]\n",
      " ...\n",
      " [-0.15662876 -0.15662876 -0.15662876 ... -0.15943459 -0.15943459\n",
      "  -0.15943459]\n",
      " [-0.21327363 -0.21327363 -0.21327363 ... -0.21343753 -0.21343753\n",
      "  -0.21343753]\n",
      " [-0.14723209 -0.14723209 -0.14723209 ... -0.1531006  -0.1531006\n",
      "  -0.1531006 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.18078542 -0.18078542 -0.18078542 ... -0.19088998 -0.19088998\n",
      "  -0.19088998]\n",
      " [-0.17433868 -0.17433868 -0.17433868 ... -0.20071343 -0.20071343\n",
      "  -0.20071343]\n",
      " [-0.15147808 -0.15147808 -0.15147808 ... -0.18063389 -0.18063389\n",
      "  -0.18063389]\n",
      " ...\n",
      " [-0.16770345 -0.16770345 -0.16770345 ... -0.1810242  -0.1810242\n",
      "  -0.1810242 ]\n",
      " [-0.1797665  -0.1797665  -0.1797665  ... -0.16856408 -0.16856408\n",
      "  -0.16856408]\n",
      " [-0.18434817 -0.18434817 -0.18434817 ... -0.18732752 -0.18732752\n",
      "  -0.18732752]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.16973004 -0.16973004 -0.16973004 ... -0.1768957  -0.1768957\n",
      "  -0.1768957 ]\n",
      " [-0.16318175 -0.16318175 -0.16318175 ... -0.17351569 -0.17351569\n",
      "  -0.17351569]\n",
      " [-0.19722465 -0.19722465 -0.19722465 ... -0.12657231 -0.12657231\n",
      "  -0.12657231]\n",
      " ...\n",
      " [-0.15387037 -0.15387037 -0.15387037 ... -0.21869072 -0.21869072\n",
      "  -0.21869072]\n",
      " [-0.15910086 -0.15910086 -0.15910086 ... -0.19000584 -0.19000584\n",
      "  -0.19000584]\n",
      " [-0.1812448  -0.1812448  -0.1812448  ... -0.14500198 -0.14500198\n",
      "  -0.14500198]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15256181 -0.15256181 -0.15256181 ... -0.19033761 -0.19033761\n",
      "  -0.19033761]\n",
      " [-0.16779302 -0.16779302 -0.16779302 ... -0.15475559 -0.15475559\n",
      "  -0.15475559]\n",
      " [-0.18739459 -0.18739459 -0.18739459 ... -0.17936102 -0.17936102\n",
      "  -0.17936102]\n",
      " ...\n",
      " [-0.16340604 -0.16340604 -0.16340604 ... -0.15082543 -0.15082543\n",
      "  -0.15082543]\n",
      " [-0.17209578 -0.17209578 -0.17209578 ... -0.1732738  -0.1732738\n",
      "  -0.1732738 ]\n",
      " [-0.15773182 -0.15773182 -0.15773182 ... -0.19281384 -0.19281384\n",
      "  -0.19281384]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3030  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.14089397 -0.14089397 -0.14089397 ... -0.17001638 -0.17001638\n",
      "  -0.17001638]\n",
      " [-0.15974268 -0.15974268 -0.15974268 ... -0.13828069 -0.13828069\n",
      "  -0.13828069]\n",
      " [-0.16816837 -0.16816837 -0.16816837 ... -0.15532939 -0.15532939\n",
      "  -0.15532939]\n",
      " ...\n",
      " [-0.18909207 -0.18909207 -0.18909207 ... -0.16602454 -0.16602454\n",
      "  -0.16602454]\n",
      " [-0.15734053 -0.15734053 -0.15734053 ... -0.16888103 -0.16888103\n",
      "  -0.16888103]\n",
      " [-0.20915152 -0.20915152 -0.20915152 ... -0.15433106 -0.15433106\n",
      "  -0.15433106]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16203427 -0.16203427 -0.16203427 ... -0.14582303 -0.14582303\n",
      "  -0.14582303]\n",
      " [-0.16074306 -0.16074306 -0.16074306 ... -0.17849278 -0.17849278\n",
      "  -0.17849278]\n",
      " [-0.17966071 -0.17966071 -0.17966071 ... -0.1799534  -0.1799534\n",
      "  -0.1799534 ]\n",
      " ...\n",
      " [-0.18194363 -0.18194363 -0.18194363 ... -0.17365296 -0.17365296\n",
      "  -0.17365296]\n",
      " [-0.18584444 -0.18584444 -0.18584444 ... -0.14807317 -0.14807317\n",
      "  -0.14807317]\n",
      " [-0.15786222 -0.15786222 -0.15786222 ... -0.15262929 -0.15262929\n",
      "  -0.15262929]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15089002 -0.15089002 -0.15089002 ... -0.16443917 -0.16443917\n",
      "  -0.16443917]\n",
      " [-0.22029826 -0.22029826 -0.22029826 ... -0.1556536  -0.1556536\n",
      "  -0.1556536 ]\n",
      " [-0.12035084 -0.12035084 -0.12035084 ... -0.16665089 -0.16665089\n",
      "  -0.16665089]\n",
      " ...\n",
      " [-0.14903027 -0.14903027 -0.14903027 ... -0.14666307 -0.14666307\n",
      "  -0.14666307]\n",
      " [-0.17025895 -0.17025895 -0.17025895 ... -0.14784567 -0.14784567\n",
      "  -0.14784567]\n",
      " [-0.12592731 -0.12592731 -0.12592731 ... -0.16504316 -0.16504316\n",
      "  -0.16504316]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14870544 -0.14870544 -0.14870544 ... -0.20263553 -0.20263553\n",
      "  -0.20263553]\n",
      " [-0.17210874 -0.17210874 -0.17210874 ... -0.17968005 -0.17968005\n",
      "  -0.17968005]\n",
      " [-0.16557899 -0.16557899 -0.16557899 ... -0.09692962 -0.09692962\n",
      "  -0.09692962]\n",
      " ...\n",
      " [-0.18134524 -0.18134524 -0.18134524 ... -0.1728132  -0.1728132\n",
      "  -0.1728132 ]\n",
      " [-0.1848281  -0.1848281  -0.1848281  ... -0.12119983 -0.12119983\n",
      "  -0.12119983]\n",
      " [-0.18706444 -0.18706444 -0.18706444 ... -0.17588103 -0.17588103\n",
      "  -0.17588103]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.19174416 -0.19174416 -0.19174416 ... -0.1662325  -0.1662325\n",
      "  -0.1662325 ]\n",
      " [-0.16288634 -0.16288634 -0.16288634 ... -0.17257543 -0.17257543\n",
      "  -0.17257543]\n",
      " [-0.16131045 -0.16131045 -0.16131045 ... -0.16490233 -0.16490233\n",
      "  -0.16490233]\n",
      " ...\n",
      " [-0.15292597 -0.15292597 -0.15292597 ... -0.1585483  -0.1585483\n",
      "  -0.1585483 ]\n",
      " [-0.16649374 -0.16649374 -0.16649374 ... -0.16409777 -0.16409777\n",
      "  -0.16409777]\n",
      " [-0.12612556 -0.12612556 -0.12612556 ... -0.1791183  -0.1791183\n",
      "  -0.1791183 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.151345   -0.151345   -0.151345   ... -0.1565192  -0.1565192\n",
      "  -0.1565192 ]\n",
      " [-0.17895174 -0.17895174 -0.17895174 ... -0.1429563  -0.1429563\n",
      "  -0.1429563 ]\n",
      " [-0.1588629  -0.1588629  -0.1588629  ... -0.15774553 -0.15774553\n",
      "  -0.15774553]\n",
      " ...\n",
      " [-0.18394932 -0.18394932 -0.18394932 ... -0.13967031 -0.13967031\n",
      "  -0.13967031]\n",
      " [-0.14761049 -0.14761049 -0.14761049 ... -0.14577779 -0.14577779\n",
      "  -0.14577779]\n",
      " [-0.18036704 -0.18036704 -0.18036704 ... -0.1818994  -0.1818994\n",
      "  -0.1818994 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.16936521 -0.16936521 -0.16936521 ... -0.15088719 -0.15088719\n",
      "  -0.15088719]\n",
      " [-0.18375114 -0.18375114 -0.18375114 ... -0.14080298 -0.14080298\n",
      "  -0.14080298]\n",
      " [-0.1830861  -0.1830861  -0.1830861  ... -0.16501027 -0.16501027\n",
      "  -0.16501027]\n",
      " ...\n",
      " [-0.13832194 -0.13832194 -0.13832194 ... -0.1578067  -0.1578067\n",
      "  -0.1578067 ]\n",
      " [-0.19300634 -0.19300634 -0.19300634 ... -0.15322928 -0.15322928\n",
      "  -0.15322928]\n",
      " [-0.13392559 -0.13392559 -0.13392559 ... -0.15216798 -0.15216798\n",
      "  -0.15216798]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.17150119 -0.17150119 -0.17150119 ... -0.22015244 -0.22015244\n",
      "  -0.22015244]\n",
      " [-0.19606976 -0.19606976 -0.19606976 ... -0.15911697 -0.15911697\n",
      "  -0.15911697]\n",
      " [-0.15501156 -0.15501156 -0.15501156 ... -0.19138    -0.19138\n",
      "  -0.19138   ]\n",
      " ...\n",
      " [-0.21952231 -0.21952231 -0.21952231 ... -0.1229647  -0.1229647\n",
      "  -0.1229647 ]\n",
      " [-0.16636007 -0.16636007 -0.16636007 ... -0.20249411 -0.20249411\n",
      "  -0.20249411]\n",
      " [-0.17704488 -0.17704488 -0.17704488 ... -0.14604375 -0.14604375\n",
      "  -0.14604375]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3131  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.12719709 -0.12719709 -0.12719709 ... -0.1412419  -0.1412419\n",
      "  -0.1412419 ]\n",
      " [-0.17969316 -0.17969316 -0.17969316 ... -0.20229405 -0.20229405\n",
      "  -0.20229405]\n",
      " [-0.16765615 -0.16765615 -0.16765615 ... -0.15498263 -0.15498263\n",
      "  -0.15498263]\n",
      " ...\n",
      " [-0.1480453  -0.1480453  -0.1480453  ... -0.12433743 -0.12433743\n",
      "  -0.12433743]\n",
      " [-0.1405828  -0.1405828  -0.1405828  ... -0.14538741 -0.14538741\n",
      "  -0.14538741]\n",
      " [-0.16877557 -0.16877557 -0.16877557 ... -0.1536142  -0.1536142\n",
      "  -0.1536142 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.12925762 -0.12925762 -0.12925762 ... -0.13614383 -0.13614383\n",
      "  -0.13614383]\n",
      " [-0.17999989 -0.17999989 -0.17999989 ... -0.16180429 -0.16180429\n",
      "  -0.16180429]\n",
      " [-0.1802074  -0.1802074  -0.1802074  ... -0.15041023 -0.15041023\n",
      "  -0.15041023]\n",
      " ...\n",
      " [-0.16639516 -0.16639516 -0.16639516 ... -0.15141232 -0.15141232\n",
      "  -0.15141232]\n",
      " [-0.16608003 -0.16608003 -0.16608003 ... -0.17930359 -0.17930359\n",
      "  -0.17930359]\n",
      " [-0.16378644 -0.16378644 -0.16378644 ... -0.17830718 -0.17830718\n",
      "  -0.17830718]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15311973 -0.15311973 -0.15311973 ... -0.13334765 -0.13334765\n",
      "  -0.13334765]\n",
      " [-0.17652759 -0.17652759 -0.17652759 ... -0.13003832 -0.13003832\n",
      "  -0.13003832]\n",
      " [-0.16611384 -0.16611384 -0.16611384 ... -0.1624502  -0.1624502\n",
      "  -0.1624502 ]\n",
      " ...\n",
      " [-0.13178498 -0.13178498 -0.13178498 ... -0.1660515  -0.1660515\n",
      "  -0.1660515 ]\n",
      " [-0.17129013 -0.17129013 -0.17129013 ... -0.15426874 -0.15426874\n",
      "  -0.15426874]\n",
      " [-0.15801924 -0.15801924 -0.15801924 ... -0.1532332  -0.1532332\n",
      "  -0.1532332 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15211359 -0.15211359 -0.15211359 ... -0.17727786 -0.17727786\n",
      "  -0.17727786]\n",
      " [-0.14768806 -0.14768806 -0.14768806 ... -0.17439726 -0.17439726\n",
      "  -0.17439726]\n",
      " [-0.15841717 -0.15841717 -0.15841717 ... -0.14402884 -0.14402884\n",
      "  -0.14402884]\n",
      " ...\n",
      " [-0.1579561  -0.1579561  -0.1579561  ... -0.1509242  -0.1509242\n",
      "  -0.1509242 ]\n",
      " [-0.10757394 -0.10757394 -0.10757394 ... -0.1509749  -0.1509749\n",
      "  -0.1509749 ]\n",
      " [-0.13895404 -0.13895404 -0.13895404 ... -0.1808232  -0.1808232\n",
      "  -0.1808232 ]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13460776 -0.13460776 -0.13460776 ... -0.18626653 -0.18626653\n",
      "  -0.18626653]\n",
      " [-0.16811523 -0.16811523 -0.16811523 ... -0.13699307 -0.13699307\n",
      "  -0.13699307]\n",
      " [-0.12896591 -0.12896591 -0.12896591 ... -0.18561368 -0.18561368\n",
      "  -0.18561368]\n",
      " ...\n",
      " [-0.15955155 -0.15955155 -0.15955155 ... -0.13931665 -0.13931665\n",
      "  -0.13931665]\n",
      " [-0.18427731 -0.18427731 -0.18427731 ... -0.14722992 -0.14722992\n",
      "  -0.14722992]\n",
      " [-0.15045872 -0.15045872 -0.15045872 ... -0.16454871 -0.16454871\n",
      "  -0.16454871]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16827816 -0.16827816 -0.16827816 ... -0.15327618 -0.15327618\n",
      "  -0.15327618]\n",
      " [-0.1559475  -0.1559475  -0.1559475  ... -0.16208616 -0.16208616\n",
      "  -0.16208616]\n",
      " [-0.16789815 -0.16789815 -0.16789815 ... -0.13894266 -0.13894266\n",
      "  -0.13894266]\n",
      " ...\n",
      " [-0.15434651 -0.15434651 -0.15434651 ... -0.15162222 -0.15162222\n",
      "  -0.15162222]\n",
      " [-0.1427375  -0.1427375  -0.1427375  ... -0.18014425 -0.18014425\n",
      "  -0.18014425]\n",
      " [-0.14475071 -0.14475071 -0.14475071 ... -0.14711247 -0.14711247\n",
      "  -0.14711247]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1609096  -0.1609096  -0.1609096  ... -0.2088098  -0.2088098\n",
      "  -0.2088098 ]\n",
      " [-0.1767466  -0.1767466  -0.1767466  ... -0.14660159 -0.14660159\n",
      "  -0.14660159]\n",
      " [-0.15466553 -0.15466553 -0.15466553 ... -0.16851655 -0.16851655\n",
      "  -0.16851655]\n",
      " ...\n",
      " [-0.12926655 -0.12926655 -0.12926655 ... -0.12213868 -0.12213868\n",
      "  -0.12213868]\n",
      " [-0.17046681 -0.17046681 -0.17046681 ... -0.1496186  -0.1496186\n",
      "  -0.1496186 ]\n",
      " [-0.14189246 -0.14189246 -0.14189246 ... -0.18003681 -0.18003681\n",
      "  -0.18003681]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.11294555 -0.11294555 -0.11294555 ... -0.13783498 -0.13783498\n",
      "  -0.13783498]\n",
      " [-0.17725372 -0.17725372 -0.17725372 ... -0.16351245 -0.16351245\n",
      "  -0.16351245]\n",
      " [-0.14229375 -0.14229375 -0.14229375 ... -0.16453701 -0.16453701\n",
      "  -0.16453701]\n",
      " ...\n",
      " [-0.13425629 -0.13425629 -0.13425629 ... -0.15599352 -0.15599352\n",
      "  -0.15599352]\n",
      " [-0.14356703 -0.14356703 -0.14356703 ... -0.17561103 -0.17561103\n",
      "  -0.17561103]\n",
      " [-0.16609801 -0.16609801 -0.16609801 ... -0.17483851 -0.17483851\n",
      "  -0.17483851]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3232  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1565886  -0.1565886  -0.1565886  ... -0.16902494 -0.16902494\n",
      "  -0.16902494]\n",
      " [-0.14739813 -0.14739813 -0.14739813 ... -0.12419644 -0.12419644\n",
      "  -0.12419644]\n",
      " [-0.11263407 -0.11263407 -0.11263407 ... -0.1530827  -0.1530827\n",
      "  -0.1530827 ]\n",
      " ...\n",
      " [-0.13992335 -0.13992335 -0.13992335 ... -0.13432045 -0.13432045\n",
      "  -0.13432045]\n",
      " [-0.14071926 -0.14071926 -0.14071926 ... -0.12428801 -0.12428801\n",
      "  -0.12428801]\n",
      " [-0.10064483 -0.10064483 -0.10064483 ... -0.13213424 -0.13213424\n",
      "  -0.13213424]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.11289354 -0.11289354 -0.11289354 ... -0.14271553 -0.14271553\n",
      "  -0.14271553]\n",
      " [-0.12593958 -0.12593958 -0.12593958 ... -0.19936348 -0.19936348\n",
      "  -0.19936348]\n",
      " [-0.12761782 -0.12761782 -0.12761782 ... -0.13197456 -0.13197456\n",
      "  -0.13197456]\n",
      " ...\n",
      " [-0.15315863 -0.15315863 -0.15315863 ... -0.14794946 -0.14794946\n",
      "  -0.14794946]\n",
      " [-0.120086   -0.120086   -0.120086   ... -0.162556   -0.162556\n",
      "  -0.162556  ]\n",
      " [-0.141009   -0.141009   -0.141009   ... -0.14955112 -0.14955112\n",
      "  -0.14955112]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1347287  -0.1347287  -0.1347287  ... -0.15741453 -0.15741453\n",
      "  -0.15741453]\n",
      " [-0.14633787 -0.14633787 -0.14633787 ... -0.14526743 -0.14526743\n",
      "  -0.14526743]\n",
      " [-0.13609885 -0.13609885 -0.13609885 ... -0.12804785 -0.12804785\n",
      "  -0.12804785]\n",
      " ...\n",
      " [-0.17668226 -0.17668226 -0.17668226 ... -0.10265133 -0.10265133\n",
      "  -0.10265133]\n",
      " [-0.13681184 -0.13681184 -0.13681184 ... -0.13768253 -0.13768253\n",
      "  -0.13768253]\n",
      " [-0.17027432 -0.17027432 -0.17027432 ... -0.1622708  -0.1622708\n",
      "  -0.1622708 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14333616 -0.14333616 -0.14333616 ... -0.1633091  -0.1633091\n",
      "  -0.1633091 ]\n",
      " [-0.1315499  -0.1315499  -0.1315499  ... -0.1686455  -0.1686455\n",
      "  -0.1686455 ]\n",
      " [-0.1425703  -0.1425703  -0.1425703  ... -0.14076264 -0.14076264\n",
      "  -0.14076264]\n",
      " ...\n",
      " [-0.13368575 -0.13368575 -0.13368575 ... -0.17212872 -0.17212872\n",
      "  -0.17212872]\n",
      " [-0.17739052 -0.17739052 -0.17739052 ... -0.15438658 -0.15438658\n",
      "  -0.15438658]\n",
      " [-0.1579446  -0.1579446  -0.1579446  ... -0.12829016 -0.12829016\n",
      "  -0.12829016]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16815151 -0.16815151 -0.16815151 ... -0.1458141  -0.1458141\n",
      "  -0.1458141 ]\n",
      " [-0.13497877 -0.13497877 -0.13497877 ... -0.13760954 -0.13760954\n",
      "  -0.13760954]\n",
      " [-0.15989488 -0.15989488 -0.15989488 ... -0.14937398 -0.14937398\n",
      "  -0.14937398]\n",
      " ...\n",
      " [-0.17722663 -0.17722663 -0.17722663 ... -0.1427409  -0.1427409\n",
      "  -0.1427409 ]\n",
      " [-0.147671   -0.147671   -0.147671   ... -0.13292259 -0.13292259\n",
      "  -0.13292259]\n",
      " [-0.16486892 -0.16486892 -0.16486892 ... -0.1675607  -0.1675607\n",
      "  -0.1675607 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13432895 -0.13432895 -0.13432895 ... -0.13488185 -0.13488185\n",
      "  -0.13488185]\n",
      " [-0.07660555 -0.07660555 -0.07660555 ... -0.12876102 -0.12876102\n",
      "  -0.12876102]\n",
      " [-0.11597651 -0.11597651 -0.11597651 ... -0.14008886 -0.14008886\n",
      "  -0.14008886]\n",
      " ...\n",
      " [-0.16307572 -0.16307572 -0.16307572 ... -0.14733616 -0.14733616\n",
      "  -0.14733616]\n",
      " [-0.15181512 -0.15181512 -0.15181512 ... -0.1278985  -0.1278985\n",
      "  -0.1278985 ]\n",
      " [-0.1458592  -0.1458592  -0.1458592  ... -0.1342725  -0.1342725\n",
      "  -0.1342725 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13156961 -0.13156961 -0.13156961 ... -0.14779699 -0.14779699\n",
      "  -0.14779699]\n",
      " [-0.12943737 -0.12943737 -0.12943737 ... -0.13617337 -0.13617337\n",
      "  -0.13617337]\n",
      " [-0.13459834 -0.13459834 -0.13459834 ... -0.16534615 -0.16534615\n",
      "  -0.16534615]\n",
      " ...\n",
      " [-0.12188776 -0.12188776 -0.12188776 ... -0.16774464 -0.16774464\n",
      "  -0.16774464]\n",
      " [-0.16152133 -0.16152133 -0.16152133 ... -0.12146777 -0.12146777\n",
      "  -0.12146777]\n",
      " [-0.16438724 -0.16438724 -0.16438724 ... -0.15915196 -0.15915196\n",
      "  -0.15915196]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.13750595 -0.13750595 -0.13750595 ... -0.1588168  -0.1588168\n",
      "  -0.1588168 ]\n",
      " [-0.16364458 -0.16364458 -0.16364458 ... -0.14622986 -0.14622986\n",
      "  -0.14622986]\n",
      " [-0.1605919  -0.1605919  -0.1605919  ... -0.15322779 -0.15322779\n",
      "  -0.15322779]\n",
      " ...\n",
      " [-0.16285795 -0.16285795 -0.16285795 ... -0.14335074 -0.14335074\n",
      "  -0.14335074]\n",
      " [-0.14170463 -0.14170463 -0.14170463 ... -0.15667479 -0.15667479\n",
      "  -0.15667479]\n",
      " [-0.1622192  -0.1622192  -0.1622192  ... -0.14696257 -0.14696257\n",
      "  -0.14696257]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3333  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16786699 -0.16786699 -0.16786699 ... -0.12432513 -0.12432513\n",
      "  -0.12432513]\n",
      " [-0.12023001 -0.12023001 -0.12023001 ... -0.14293493 -0.14293493\n",
      "  -0.14293493]\n",
      " [-0.20438741 -0.20438741 -0.20438741 ... -0.16099751 -0.16099751\n",
      "  -0.16099751]\n",
      " ...\n",
      " [-0.14655146 -0.14655146 -0.14655146 ... -0.13345116 -0.13345116\n",
      "  -0.13345116]\n",
      " [-0.1622978  -0.1622978  -0.1622978  ... -0.14028215 -0.14028215\n",
      "  -0.14028215]\n",
      " [-0.15073194 -0.15073194 -0.15073194 ... -0.13066399 -0.13066399\n",
      "  -0.13066399]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13448074 -0.13448074 -0.13448074 ... -0.13804072 -0.13804072\n",
      "  -0.13804072]\n",
      " [-0.17854105 -0.17854105 -0.17854105 ... -0.08343516 -0.08343516\n",
      "  -0.08343516]\n",
      " [-0.11044522 -0.11044522 -0.11044522 ... -0.16360644 -0.16360644\n",
      "  -0.16360644]\n",
      " ...\n",
      " [-0.14003728 -0.14003728 -0.14003728 ... -0.11871774 -0.11871774\n",
      "  -0.11871774]\n",
      " [-0.18174577 -0.18174577 -0.18174577 ... -0.14629582 -0.14629582\n",
      "  -0.14629582]\n",
      " [-0.12255219 -0.12255219 -0.12255219 ... -0.14129223 -0.14129223\n",
      "  -0.14129223]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20550074 -0.20550074 -0.20550074 ... -0.12456988 -0.12456988\n",
      "  -0.12456988]\n",
      " [-0.1732966  -0.1732966  -0.1732966  ... -0.1586803  -0.1586803\n",
      "  -0.1586803 ]\n",
      " [-0.1078537  -0.1078537  -0.1078537  ... -0.16189526 -0.16189526\n",
      "  -0.16189526]\n",
      " ...\n",
      " [-0.15235606 -0.15235606 -0.15235606 ... -0.1487467  -0.1487467\n",
      "  -0.1487467 ]\n",
      " [-0.14056736 -0.14056736 -0.14056736 ... -0.1428244  -0.1428244\n",
      "  -0.1428244 ]\n",
      " [-0.1305733  -0.1305733  -0.1305733  ... -0.1750105  -0.1750105\n",
      "  -0.1750105 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.10172161 -0.10172161 -0.10172161 ... -0.16871345 -0.16871345\n",
      "  -0.16871345]\n",
      " [-0.11605466 -0.11605466 -0.11605466 ... -0.13858579 -0.13858579\n",
      "  -0.13858579]\n",
      " [-0.13718511 -0.13718511 -0.13718511 ... -0.14906159 -0.14906159\n",
      "  -0.14906159]\n",
      " ...\n",
      " [-0.12705714 -0.12705714 -0.12705714 ... -0.12975705 -0.12975705\n",
      "  -0.12975705]\n",
      " [-0.12793632 -0.12793632 -0.12793632 ... -0.13761675 -0.13761675\n",
      "  -0.13761675]\n",
      " [-0.11605466 -0.11605466 -0.11605466 ... -0.15760556 -0.15760556\n",
      "  -0.15760556]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.15179467 -0.15179467 -0.15179467 ... -0.15955648 -0.15955648\n",
      "  -0.15955648]\n",
      " [-0.18055835 -0.18055835 -0.18055835 ... -0.11600896 -0.11600896\n",
      "  -0.11600896]\n",
      " [-0.10884997 -0.10884997 -0.10884997 ... -0.17776674 -0.17776674\n",
      "  -0.17776674]\n",
      " ...\n",
      " [-0.13852088 -0.13852088 -0.13852088 ... -0.16043793 -0.16043793\n",
      "  -0.16043793]\n",
      " [-0.12575504 -0.12575504 -0.12575504 ... -0.14956328 -0.14956328\n",
      "  -0.14956328]\n",
      " [-0.14792947 -0.14792947 -0.14792947 ... -0.12237388 -0.12237388\n",
      "  -0.12237388]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.15113187 -0.15113187 -0.15113187 ... -0.12512329 -0.12512329\n",
      "  -0.12512329]\n",
      " [-0.13550979 -0.13550979 -0.13550979 ... -0.14200753 -0.14200753\n",
      "  -0.14200753]\n",
      " [-0.13727276 -0.13727276 -0.13727276 ... -0.16642137 -0.16642137\n",
      "  -0.16642137]\n",
      " ...\n",
      " [-0.12726855 -0.12726855 -0.12726855 ... -0.15881905 -0.15881905\n",
      "  -0.15881905]\n",
      " [-0.13652496 -0.13652496 -0.13652496 ... -0.13823017 -0.13823017\n",
      "  -0.13823017]\n",
      " [-0.1577133  -0.1577133  -0.1577133  ... -0.1536934  -0.1536934\n",
      "  -0.1536934 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.17524269 -0.17524269 -0.17524269 ... -0.11193538 -0.11193538\n",
      "  -0.11193538]\n",
      " [-0.13083082 -0.13083082 -0.13083082 ... -0.1431812  -0.1431812\n",
      "  -0.1431812 ]\n",
      " [-0.15036783 -0.15036783 -0.15036783 ... -0.1515489  -0.1515489\n",
      "  -0.1515489 ]\n",
      " ...\n",
      " [-0.12573297 -0.12573297 -0.12573297 ... -0.15608615 -0.15608615\n",
      "  -0.15608615]\n",
      " [-0.16913831 -0.16913831 -0.16913831 ... -0.15669185 -0.15669185\n",
      "  -0.15669185]\n",
      " [-0.13645291 -0.13645291 -0.13645291 ... -0.16668415 -0.16668415\n",
      "  -0.16668415]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15551923 -0.15551923 -0.15551923 ... -0.08175788 -0.08175788\n",
      "  -0.08175788]\n",
      " [-0.13804501 -0.13804501 -0.13804501 ... -0.12601696 -0.12601696\n",
      "  -0.12601696]\n",
      " [-0.14003728 -0.14003728 -0.14003728 ... -0.12830302 -0.12830302\n",
      "  -0.12830302]\n",
      " ...\n",
      " [-0.11073717 -0.11073717 -0.11073717 ... -0.14511503 -0.14511503\n",
      "  -0.14511503]\n",
      " [-0.11889408 -0.11889408 -0.11889408 ... -0.14069523 -0.14069523\n",
      "  -0.14069523]\n",
      " [-0.13057624 -0.13057624 -0.13057624 ... -0.10603548 -0.10603548\n",
      "  -0.10603548]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3434  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13456732 -0.13456732 -0.13456732 ... -0.15443516 -0.15443516\n",
      "  -0.15443516]\n",
      " [-0.10452057 -0.10452057 -0.10452057 ... -0.10825461 -0.10825461\n",
      "  -0.10825461]\n",
      " [-0.11846714 -0.11846714 -0.11846714 ... -0.10796934 -0.10796934\n",
      "  -0.10796934]\n",
      " ...\n",
      " [-0.17749336 -0.17749336 -0.17749336 ... -0.17231855 -0.17231855\n",
      "  -0.17231855]\n",
      " [-0.12192442 -0.12192442 -0.12192442 ... -0.05856589 -0.05856589\n",
      "  -0.05856589]\n",
      " [-0.14656523 -0.14656523 -0.14656523 ... -0.12245967 -0.12245967\n",
      "  -0.12245967]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.09632822 -0.09632822 -0.09632822 ... -0.14483076 -0.14483076\n",
      "  -0.14483076]\n",
      " [-0.1091676  -0.1091676  -0.1091676  ... -0.1472805  -0.1472805\n",
      "  -0.1472805 ]\n",
      " [-0.16943567 -0.16943567 -0.16943567 ... -0.1434523  -0.1434523\n",
      "  -0.1434523 ]\n",
      " ...\n",
      " [-0.13530068 -0.13530068 -0.13530068 ... -0.12187108 -0.12187108\n",
      "  -0.12187108]\n",
      " [-0.13225706 -0.13225706 -0.13225706 ... -0.15744056 -0.15744056\n",
      "  -0.15744056]\n",
      " [-0.16659372 -0.16659372 -0.16659372 ... -0.12503713 -0.12503713\n",
      "  -0.12503713]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14522235 -0.14522235 -0.14522235 ... -0.1533368  -0.1533368\n",
      "  -0.1533368 ]\n",
      " [-0.14943798 -0.14943798 -0.14943798 ... -0.15821162 -0.15821162\n",
      "  -0.15821162]\n",
      " [-0.14536527 -0.14536527 -0.14536527 ... -0.1410813  -0.1410813\n",
      "  -0.1410813 ]\n",
      " ...\n",
      " [-0.11644541 -0.11644541 -0.11644541 ... -0.12001754 -0.12001754\n",
      "  -0.12001754]\n",
      " [-0.12837133 -0.12837133 -0.12837133 ... -0.11800954 -0.11800954\n",
      "  -0.11800954]\n",
      " [-0.12038189 -0.12038189 -0.12038189 ... -0.11567429 -0.11567429\n",
      "  -0.11567429]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14829147 -0.14829147 -0.14829147 ... -0.09697851 -0.09697851\n",
      "  -0.09697851]\n",
      " [-0.10536797 -0.10536797 -0.10536797 ... -0.16726488 -0.16726488\n",
      "  -0.16726488]\n",
      " [-0.12835777 -0.12835777 -0.12835777 ... -0.17094886 -0.17094886\n",
      "  -0.17094886]\n",
      " ...\n",
      " [-0.15998146 -0.15998146 -0.15998146 ... -0.13198906 -0.13198906\n",
      "  -0.13198906]\n",
      " [-0.14269827 -0.14269827 -0.14269827 ... -0.15083882 -0.15083882\n",
      "  -0.15083882]\n",
      " [-0.1598942  -0.1598942  -0.1598942  ... -0.14486961 -0.14486961\n",
      "  -0.14486961]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.14747244 -0.14747244 -0.14747244 ... -0.1309759  -0.1309759\n",
      "  -0.1309759 ]\n",
      " [-0.13157403 -0.13157403 -0.13157403 ... -0.13023405 -0.13023405\n",
      "  -0.13023405]\n",
      " [-0.1297859  -0.1297859  -0.1297859  ... -0.11079148 -0.11079148\n",
      "  -0.11079148]\n",
      " ...\n",
      " [-0.14686443 -0.14686443 -0.14686443 ... -0.14289051 -0.14289051\n",
      "  -0.14289051]\n",
      " [-0.11602416 -0.11602416 -0.11602416 ... -0.05856589 -0.05856589\n",
      "  -0.05856589]\n",
      " [-0.16118762 -0.16118762 -0.16118762 ... -0.13944644 -0.13944644\n",
      "  -0.13944644]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.09866805 -0.09866805 -0.09866805 ... -0.11413323 -0.11413323\n",
      "  -0.11413323]\n",
      " [-0.1725307  -0.1725307  -0.1725307  ... -0.16141652 -0.16141652\n",
      "  -0.16141652]\n",
      " [-0.13714427 -0.13714427 -0.13714427 ... -0.13591951 -0.13591951\n",
      "  -0.13591951]\n",
      " ...\n",
      " [-0.1346918  -0.1346918  -0.1346918  ... -0.13579184 -0.13579184\n",
      "  -0.13579184]\n",
      " [-0.15924418 -0.15924418 -0.15924418 ... -0.16946225 -0.16946225\n",
      "  -0.16946225]\n",
      " [-0.12373075 -0.12373075 -0.12373075 ... -0.15191387 -0.15191387\n",
      "  -0.15191387]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14047202 -0.14047202 -0.14047202 ... -0.14059772 -0.14059772\n",
      "  -0.14059772]\n",
      " [-0.24581327 -0.24581327 -0.24581327 ... -0.15396048 -0.15396048\n",
      "  -0.15396048]\n",
      " [-0.13342126 -0.13342126 -0.13342126 ... -0.13012657 -0.13012657\n",
      "  -0.13012657]\n",
      " ...\n",
      " [-0.15228522 -0.15228522 -0.15228522 ... -0.12704153 -0.12704153\n",
      "  -0.12704153]\n",
      " [-0.15143487 -0.15143487 -0.15143487 ... -0.11653686 -0.11653686\n",
      "  -0.11653686]\n",
      " [-0.15302896 -0.15302896 -0.15302896 ... -0.1373566  -0.1373566\n",
      "  -0.1373566 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.11793421 -0.11793421 -0.11793421 ... -0.1214875  -0.1214875\n",
      "  -0.1214875 ]\n",
      " [-0.16125517 -0.16125517 -0.16125517 ... -0.15006259 -0.15006259\n",
      "  -0.15006259]\n",
      " [-0.13549308 -0.13549308 -0.13549308 ... -0.14521447 -0.14521447\n",
      "  -0.14521447]\n",
      " ...\n",
      " [-0.12885591 -0.12885591 -0.12885591 ... -0.12928495 -0.12928495\n",
      "  -0.12928495]\n",
      " [-0.16101258 -0.16101258 -0.16101258 ... -0.2075745  -0.2075745\n",
      "  -0.2075745 ]\n",
      " [-0.12507053 -0.12507053 -0.12507053 ... -0.13073197 -0.13073197\n",
      "  -0.13073197]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3535  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13042389 -0.13042389 -0.13042389 ... -0.16120575 -0.16120575\n",
      "  -0.16120575]\n",
      " [-0.1140679  -0.1140679  -0.1140679  ... -0.1793801  -0.1793801\n",
      "  -0.1793801 ]\n",
      " [-0.15151946 -0.15151946 -0.15151946 ... -0.13189353 -0.13189353\n",
      "  -0.13189353]\n",
      " ...\n",
      " [-0.11810621 -0.11810621 -0.11810621 ... -0.12598315 -0.12598315\n",
      "  -0.12598315]\n",
      " [-0.1140888  -0.1140888  -0.1140888  ... -0.13730173 -0.13730173\n",
      "  -0.13730173]\n",
      " [-0.1280609  -0.1280609  -0.1280609  ... -0.15322414 -0.15322414\n",
      "  -0.15322414]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.08685695 -0.08685695 -0.08685695 ... -0.13824734 -0.13824734\n",
      "  -0.13824734]\n",
      " [-0.14149283 -0.14149283 -0.14149283 ... -0.0972722  -0.0972722\n",
      "  -0.0972722 ]\n",
      " [-0.0996718  -0.0996718  -0.0996718  ... -0.11918974 -0.11918974\n",
      "  -0.11918974]\n",
      " ...\n",
      " [-0.15187223 -0.15187223 -0.15187223 ... -0.15988265 -0.15988265\n",
      "  -0.15988265]\n",
      " [-0.10215785 -0.10215785 -0.10215785 ... -0.13392131 -0.13392131\n",
      "  -0.13392131]\n",
      " [-0.11549374 -0.11549374 -0.11549374 ... -0.14812171 -0.14812171\n",
      "  -0.14812171]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.12124658 -0.12124658 -0.12124658 ... -0.13766557 -0.13766557\n",
      "  -0.13766557]\n",
      " [-0.14898978 -0.14898978 -0.14898978 ... -0.12508199 -0.12508199\n",
      "  -0.12508199]\n",
      " [-0.11298446 -0.11298446 -0.11298446 ... -0.1344096  -0.1344096\n",
      "  -0.1344096 ]\n",
      " ...\n",
      " [-0.12948906 -0.12948906 -0.12948906 ... -0.12276616 -0.12276616\n",
      "  -0.12276616]\n",
      " [-0.1082052  -0.1082052  -0.1082052  ... -0.17274304 -0.17274304\n",
      "  -0.17274304]\n",
      " [-0.12128858 -0.12128858 -0.12128858 ... -0.14868814 -0.14868814\n",
      "  -0.14868814]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.10334651 -0.10334651 -0.10334651 ... -0.12731408 -0.12731408\n",
      "  -0.12731408]\n",
      " [-0.1516147  -0.1516147  -0.1516147  ... -0.14997691 -0.14997691\n",
      "  -0.14997691]\n",
      " [-0.18005063 -0.18005063 -0.18005063 ... -0.14951757 -0.14951757\n",
      "  -0.14951757]\n",
      " ...\n",
      " [-0.12168486 -0.12168486 -0.12168486 ... -0.13384208 -0.13384208\n",
      "  -0.13384208]\n",
      " [-0.1006973  -0.1006973  -0.1006973  ... -0.12426579 -0.12426579\n",
      "  -0.12426579]\n",
      " [-0.11459538 -0.11459538 -0.11459538 ... -0.14974831 -0.14974831\n",
      "  -0.14974831]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1070858  -0.1070858  -0.1070858  ... -0.1116543  -0.1116543\n",
      "  -0.1116543 ]\n",
      " [-0.17264812 -0.17264812 -0.17264812 ... -0.16044508 -0.16044508\n",
      "  -0.16044508]\n",
      " [-0.12275766 -0.12275766 -0.12275766 ... -0.13696471 -0.13696471\n",
      "  -0.13696471]\n",
      " ...\n",
      " [-0.12987857 -0.12987857 -0.12987857 ... -0.1210803  -0.1210803\n",
      "  -0.1210803 ]\n",
      " [-0.12533084 -0.12533084 -0.12533084 ... -0.14575209 -0.14575209\n",
      "  -0.14575209]\n",
      " [-0.16174267 -0.16174267 -0.16174267 ... -0.12681219 -0.12681219\n",
      "  -0.12681219]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.15467364 -0.15467364 -0.15467364 ... -0.15297392 -0.15297392\n",
      "  -0.15297392]\n",
      " [-0.12782827 -0.12782827 -0.12782827 ... -0.10599958 -0.10599958\n",
      "  -0.10599958]\n",
      " [-0.14748351 -0.14748351 -0.14748351 ... -0.1477443  -0.1477443\n",
      "  -0.1477443 ]\n",
      " ...\n",
      " [-0.09363002 -0.09363002 -0.09363002 ... -0.14193249 -0.14193249\n",
      "  -0.14193249]\n",
      " [-0.12971857 -0.12971857 -0.12971857 ... -0.1352248  -0.1352248\n",
      "  -0.1352248 ]\n",
      " [-0.1065889  -0.1065889  -0.1065889  ... -0.11745781 -0.11745781\n",
      "  -0.11745781]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.15555145 -0.15555145 -0.15555145 ... -0.13715422 -0.13715422\n",
      "  -0.13715422]\n",
      " [-0.12356691 -0.12356691 -0.12356691 ... -0.13932261 -0.13932261\n",
      "  -0.13932261]\n",
      " [-0.18296644 -0.18296644 -0.18296644 ... -0.11947586 -0.11947586\n",
      "  -0.11947586]\n",
      " ...\n",
      " [-0.15463202 -0.15463202 -0.15463202 ... -0.14370166 -0.14370166\n",
      "  -0.14370166]\n",
      " [-0.15650143 -0.15650143 -0.15650143 ... -0.11596105 -0.11596105\n",
      "  -0.11596105]\n",
      " [-0.14271888 -0.14271888 -0.14271888 ... -0.13661465 -0.13661465\n",
      "  -0.13661465]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14254704 -0.14254704 -0.14254704 ... -0.1620166  -0.1620166\n",
      "  -0.1620166 ]\n",
      " [-0.18100819 -0.18100819 -0.18100819 ... -0.16343102 -0.16343102\n",
      "  -0.16343102]\n",
      " [-0.10871284 -0.10871284 -0.10871284 ... -0.15199263 -0.15199263\n",
      "  -0.15199263]\n",
      " ...\n",
      " [-0.11332855 -0.11332855 -0.11332855 ... -0.18051362 -0.18051362\n",
      "  -0.18051362]\n",
      " [-0.15258747 -0.15258747 -0.15258747 ... -0.13574949 -0.13574949\n",
      "  -0.13574949]\n",
      " [-0.1867454  -0.1867454  -0.1867454  ... -0.12014606 -0.12014606\n",
      "  -0.12014606]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3636  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      " [INFO]: embedding_vector\n",
      " PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.12885083 -0.12885083 -0.12885083 ... -0.12545472 -0.12545472\n",
      "  -0.12545472]\n",
      " [-0.09052454 -0.09052454 -0.09052454 ... -0.15489897 -0.15489897\n",
      "  -0.15489897]\n",
      " [-0.10301302 -0.10301302 -0.10301302 ... -0.13066089 -0.13066089\n",
      "  -0.13066089]\n",
      " ...\n",
      " [-0.14312255 -0.14312255 -0.14312255 ... -0.11650687 -0.11650687\n",
      "  -0.11650687]\n",
      " [-0.13744566 -0.13744566 -0.13744566 ... -0.13161483 -0.13161483\n",
      "  -0.13161483]\n",
      " [-0.18123242 -0.18123242 -0.18123242 ... -0.15393707 -0.15393707\n",
      "  -0.15393707]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.14850277 -0.14850277 -0.14850277 ... -0.1167239  -0.1167239\n",
      "  -0.1167239 ]\n",
      " [-0.13701148 -0.13701148 -0.13701148 ... -0.13284141 -0.13284141\n",
      "  -0.13284141]\n",
      " [-0.12825042 -0.12825042 -0.12825042 ... -0.14209066 -0.14209066\n",
      "  -0.14209066]\n",
      " ...\n",
      " [-0.12983975 -0.12983975 -0.12983975 ... -0.1385109  -0.1385109\n",
      "  -0.1385109 ]\n",
      " [-0.12772965 -0.12772965 -0.12772965 ... -0.1449002  -0.1449002\n",
      "  -0.1449002 ]\n",
      " [-0.1389724  -0.1389724  -0.1389724  ... -0.17032519 -0.17032519\n",
      "  -0.17032519]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13426757 -0.13426757 -0.13426757 ... -0.15874419 -0.15874419\n",
      "  -0.15874419]\n",
      " [-0.1577187  -0.1577187  -0.1577187  ... -0.1269598  -0.1269598\n",
      "  -0.1269598 ]\n",
      " [-0.15422975 -0.15422975 -0.15422975 ... -0.10750048 -0.10750048\n",
      "  -0.10750048]\n",
      " ...\n",
      " [-0.13934124 -0.13934124 -0.13934124 ... -0.16645561 -0.16645561\n",
      "  -0.16645561]\n",
      " [-0.17526676 -0.17526676 -0.17526676 ... -0.15791199 -0.15791199\n",
      "  -0.15791199]\n",
      " [-0.16327174 -0.16327174 -0.16327174 ... -0.16943389 -0.16943389\n",
      "  -0.16943389]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.13426942 -0.13426942 -0.13426942 ... -0.15260972 -0.15260972\n",
      "  -0.15260972]\n",
      " [-0.1451465  -0.1451465  -0.1451465  ... -0.15882014 -0.15882014\n",
      "  -0.15882014]\n",
      " [-0.14568472 -0.14568472 -0.14568472 ... -0.12397312 -0.12397312\n",
      "  -0.12397312]\n",
      " ...\n",
      " [-0.14472094 -0.14472094 -0.14472094 ... -0.14560959 -0.14560959\n",
      "  -0.14560959]\n",
      " [-0.14078566 -0.14078566 -0.14078566 ... -0.13416994 -0.13416994\n",
      "  -0.13416994]\n",
      " [-0.16058691 -0.16058691 -0.16058691 ... -0.11738507 -0.11738507\n",
      "  -0.11738507]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}\n",
      "PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.13050523 -0.13050523 -0.13050523 ... -0.12057085 -0.12057085\n",
      "  -0.12057085]\n",
      " [-0.13114573 -0.13114573 -0.13114573 ... -0.1748514  -0.1748514\n",
      "  -0.1748514 ]\n",
      " [-0.12738082 -0.12738082 -0.12738082 ... -0.12301777 -0.12301777\n",
      "  -0.12301777]\n",
      " ...\n",
      " [-0.13155879 -0.13155879 -0.13155879 ... -0.14655757 -0.14655757\n",
      "  -0.14655757]\n",
      " [-0.09117307 -0.09117307 -0.09117307 ... -0.1276281  -0.1276281\n",
      "  -0.1276281 ]\n",
      " [-0.20856729 -0.20856729 -0.20856729 ... -0.10124613 -0.10124613\n",
      "  -0.10124613]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.09517786 -0.09517786 -0.09517786 ... -0.10599483 -0.10599483\n",
      "  -0.10599483]\n",
      " [-0.13572377 -0.13572377 -0.13572377 ... -0.12271081 -0.12271081\n",
      "  -0.12271081]\n",
      " [-0.14163588 -0.14163588 -0.14163588 ... -0.1844615  -0.1844615\n",
      "  -0.1844615 ]\n",
      " ...\n",
      " [-0.1270605  -0.1270605  -0.1270605  ... -0.11611266 -0.11611266\n",
      "  -0.11611266]\n",
      " [-0.12610337 -0.12610337 -0.12610337 ... -0.15052158 -0.15052158\n",
      "  -0.15052158]\n",
      " [-0.18152963 -0.18152963 -0.18152963 ... -0.1382599  -0.1382599\n",
      "  -0.1382599 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13621019 -0.13621019 -0.13621019 ... -0.10615665 -0.10615665\n",
      "  -0.10615665]\n",
      " [-0.164194   -0.164194   -0.164194   ... -0.12670413 -0.12670413\n",
      "  -0.12670413]\n",
      " [-0.13833863 -0.13833863 -0.13833863 ... -0.18534476 -0.18534476\n",
      "  -0.18534476]\n",
      " ...\n",
      " [-0.09785162 -0.09785162 -0.09785162 ... -0.1427754  -0.1427754\n",
      "  -0.1427754 ]\n",
      " [-0.13869964 -0.13869964 -0.13869964 ... -0.14716445 -0.14716445\n",
      "  -0.14716445]\n",
      " [-0.15879509 -0.15879509 -0.15879509 ... -0.14689243 -0.14689243\n",
      "  -0.14689243]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15249106 -0.15249106 -0.15249106 ... -0.13384318 -0.13384318\n",
      "  -0.13384318]\n",
      " [-0.15715861 -0.15715861 -0.15715861 ... -0.17097239 -0.17097239\n",
      "  -0.17097239]\n",
      " [-0.1288981  -0.1288981  -0.1288981  ... -0.13762453 -0.13762453\n",
      "  -0.13762453]\n",
      " ...\n",
      " [-0.13652073 -0.13652073 -0.13652073 ... -0.14742857 -0.14742857\n",
      "  -0.14742857]\n",
      " [-0.11884924 -0.11884924 -0.11884924 ... -0.13973066 -0.13973066\n",
      "  -0.13973066]\n",
      " [-0.15960345 -0.15960345 -0.15960345 ... -0.15373829 -0.15373829\n",
      "  -0.15373829]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------------------------------------  step step   3737  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.12639283 -0.12639283 -0.12639283 ... -0.16125506 -0.16125506\n",
      "  -0.16125506]\n",
      " [-0.16674198 -0.16674198 -0.16674198 ... -0.14936584 -0.14936584\n",
      "  -0.14936584]\n",
      " [-0.07965206 -0.07965206 -0.07965206 ... -0.16007738 -0.16007738\n",
      "  -0.16007738]\n",
      " ...\n",
      " [-0.13115379 -0.13115379 -0.13115379 ... -0.13173574 -0.13173574\n",
      "  -0.13173574]\n",
      " [-0.11608835 -0.11608835 -0.11608835 ... -0.15411144 -0.15411144\n",
      "  -0.15411144]\n",
      " [-0.16858548 -0.16858548 -0.16858548 ... -0.13984138 -0.13984138\n",
      "  -0.13984138]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1600194  -0.1600194  -0.1600194  ... -0.1419123  -0.1419123\n",
      "  -0.1419123 ]\n",
      " [-0.13868913 -0.13868913 -0.13868913 ... -0.17075175 -0.17075175\n",
      "  -0.17075175]\n",
      " [-0.13906318 -0.13906318 -0.13906318 ... -0.1440883  -0.1440883\n",
      "  -0.1440883 ]\n",
      " ...\n",
      " [-0.15004958 -0.15004958 -0.15004958 ... -0.13695388 -0.13695388\n",
      "  -0.13695388]\n",
      " [-0.14480467 -0.14480467 -0.14480467 ... -0.14341122 -0.14341122\n",
      "  -0.14341122]\n",
      " [-0.14730464 -0.14730464 -0.14730464 ... -0.14242601 -0.14242601\n",
      "  -0.14242601]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.12392388 -0.12392388 -0.12392388 ... -0.1424554  -0.1424554\n",
      "  -0.1424554 ]\n",
      " [-0.13672504 -0.13672504 -0.13672504 ... -0.14604132 -0.14604132\n",
      "  -0.14604132]\n",
      " [-0.15046117 -0.15046117 -0.15046117 ... -0.13817406 -0.13817406\n",
      "  -0.13817406]\n",
      " ...\n",
      " [-0.13860628 -0.13860628 -0.13860628 ... -0.16654488 -0.16654488\n",
      "  -0.16654488]\n",
      " [-0.13192254 -0.13192254 -0.13192254 ... -0.14177972 -0.14177972\n",
      "  -0.14177972]\n",
      " [-0.15599057 -0.15599057 -0.15599057 ... -0.16168228 -0.16168228\n",
      "  -0.16168228]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.11252898 -0.11252898 -0.11252898 ... -0.13718899 -0.13718899\n",
      "  -0.13718899]\n",
      " [-0.14793429 -0.14793429 -0.14793429 ... -0.12999249 -0.12999249\n",
      "  -0.12999249]\n",
      " [-0.15925843 -0.15925843 -0.15925843 ... -0.14096412 -0.14096412\n",
      "  -0.14096412]\n",
      " ...\n",
      " [-0.14409386 -0.14409386 -0.14409386 ... -0.14399947 -0.14399947\n",
      "  -0.14399947]\n",
      " [-0.16298853 -0.16298853 -0.16298853 ... -0.12951234 -0.12951234\n",
      "  -0.12951234]\n",
      " [-0.17194451 -0.17194451 -0.17194451 ... -0.14331782 -0.14331782\n",
      "  -0.14331782]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1199398  -0.1199398  -0.1199398  ... -0.16632041 -0.16632041\n",
      "  -0.16632041]\n",
      " [-0.12410133 -0.12410133 -0.12410133 ... -0.17561166 -0.17561166\n",
      "  -0.17561166]\n",
      " [-0.13765132 -0.13765132 -0.13765132 ... -0.15518247 -0.15518247\n",
      "  -0.15518247]\n",
      " ...\n",
      " [-0.1496757  -0.1496757  -0.1496757  ... -0.17065862 -0.17065862\n",
      "  -0.17065862]\n",
      " [-0.13873594 -0.13873594 -0.13873594 ... -0.13764174 -0.13764174\n",
      "  -0.13764174]\n",
      " [-0.1435225  -0.1435225  -0.1435225  ... -0.13191284 -0.13191284\n",
      "  -0.13191284]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13431405 -0.13431405 -0.13431405 ... -0.16106918 -0.16106918\n",
      "  -0.16106918]\n",
      " [-0.15879059 -0.15879059 -0.15879059 ... -0.12588856 -0.12588856\n",
      "  -0.12588856]\n",
      " [-0.15903476 -0.15903476 -0.15903476 ... -0.13821965 -0.13821965\n",
      "  -0.13821965]\n",
      " ...\n",
      " [-0.14370994 -0.14370994 -0.14370994 ... -0.16060466 -0.16060466\n",
      "  -0.16060466]\n",
      " [-0.12784031 -0.12784031 -0.12784031 ... -0.11901534 -0.11901534\n",
      "  -0.11901534]\n",
      " [-0.18105055 -0.18105055 -0.18105055 ... -0.16158299 -0.16158299\n",
      "  -0.16158299]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.14274299 -0.14274299 -0.14274299 ... -0.14385343 -0.14385343\n",
      "  -0.14385343]\n",
      " [-0.16184863 -0.16184863 -0.16184863 ... -0.16432565 -0.16432565\n",
      "  -0.16432565]\n",
      " [-0.17879423 -0.17879423 -0.17879423 ... -0.136067   -0.136067\n",
      "  -0.136067  ]\n",
      " ...\n",
      " [-0.14909825 -0.14909825 -0.14909825 ... -0.17246509 -0.17246509\n",
      "  -0.17246509]\n",
      " [-0.13951102 -0.13951102 -0.13951102 ... -0.14788473 -0.14788473\n",
      "  -0.14788473]\n",
      " [-0.10581681 -0.10581681 -0.10581681 ... -0.179746   -0.179746\n",
      "  -0.179746  ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14771213 -0.14771213 -0.14771213 ... -0.1527952  -0.1527952\n",
      "  -0.1527952 ]\n",
      " [-0.14532202 -0.14532202 -0.14532202 ... -0.13308889 -0.13308889\n",
      "  -0.13308889]\n",
      " [-0.15458994 -0.15458994 -0.15458994 ... -0.14312297 -0.14312297\n",
      "  -0.14312297]\n",
      " ...\n",
      " [-0.16370328 -0.16370328 -0.16370328 ... -0.16159621 -0.16159621\n",
      "  -0.16159621]\n",
      " [-0.17079216 -0.17079216 -0.17079216 ... -0.18111236 -0.18111236\n",
      "  -0.18111236]\n",
      " [-0.1397759  -0.1397759  -0.1397759  ... -0.14238022 -0.14238022\n",
      "  -0.14238022]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3838  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16780905 -0.16780905 -0.16780905 ... -0.17625198 -0.17625198\n",
      "  -0.17625198]\n",
      " [-0.13975349 -0.13975349 -0.13975349 ... -0.13836461 -0.13836461\n",
      "  -0.13836461]\n",
      " [-0.12085299 -0.12085299 -0.12085299 ... -0.1586332  -0.1586332\n",
      "  -0.1586332 ]\n",
      " ...\n",
      " [-0.13169116 -0.13169116 -0.13169116 ... -0.15101323 -0.15101323\n",
      "  -0.15101323]\n",
      " [-0.14628129 -0.14628129 -0.14628129 ... -0.13259426 -0.13259426\n",
      "  -0.13259426]\n",
      " [-0.16979869 -0.16979869 -0.16979869 ... -0.12654531 -0.12654531\n",
      "  -0.12654531]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1312464  -0.1312464  -0.1312464  ... -0.13030976 -0.13030976\n",
      "  -0.13030976]\n",
      " [-0.1595909  -0.1595909  -0.1595909  ... -0.16515097 -0.16515097\n",
      "  -0.16515097]\n",
      " [-0.16183382 -0.16183382 -0.16183382 ... -0.16586408 -0.16586408\n",
      "  -0.16586408]\n",
      " ...\n",
      " [-0.09641811 -0.09641811 -0.09641811 ... -0.1906194  -0.1906194\n",
      "  -0.1906194 ]\n",
      " [-0.12407546 -0.12407546 -0.12407546 ... -0.17490768 -0.17490768\n",
      "  -0.17490768]\n",
      " [-0.15546131 -0.15546131 -0.15546131 ... -0.15400118 -0.15400118\n",
      "  -0.15400118]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1411656  -0.1411656  -0.1411656  ... -0.16703819 -0.16703819\n",
      "  -0.16703819]\n",
      " [-0.15516663 -0.15516663 -0.15516663 ... -0.16988759 -0.16988759\n",
      "  -0.16988759]\n",
      " [-0.18068904 -0.18068904 -0.18068904 ... -0.12488877 -0.12488877\n",
      "  -0.12488877]\n",
      " ...\n",
      " [-0.1399445  -0.1399445  -0.1399445  ... -0.12463865 -0.12463865\n",
      "  -0.12463865]\n",
      " [-0.15731198 -0.15731198 -0.15731198 ... -0.14513242 -0.14513242\n",
      "  -0.14513242]\n",
      " [-0.12414416 -0.12414416 -0.12414416 ... -0.16211744 -0.16211744\n",
      "  -0.16211744]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.10292963 -0.10292963 -0.10292963 ... -0.19132942 -0.19132942\n",
      "  -0.19132942]\n",
      " [-0.12897809 -0.12897809 -0.12897809 ... -0.15934324 -0.15934324\n",
      "  -0.15934324]\n",
      " [-0.1099771  -0.1099771  -0.1099771  ... -0.19072834 -0.19072834\n",
      "  -0.19072834]\n",
      " ...\n",
      " [-0.1587626  -0.1587626  -0.1587626  ... -0.13559195 -0.13559195\n",
      "  -0.13559195]\n",
      " [-0.15634184 -0.15634184 -0.15634184 ... -0.16984054 -0.16984054\n",
      "  -0.16984054]\n",
      " [-0.21025415 -0.21025415 -0.21025415 ... -0.16406289 -0.16406289\n",
      "  -0.16406289]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1500794  -0.1500794  -0.1500794  ... -0.11097371 -0.11097371\n",
      "  -0.11097371]\n",
      " [-0.18026407 -0.18026407 -0.18026407 ... -0.14487176 -0.14487176\n",
      "  -0.14487176]\n",
      " [-0.11632041 -0.11632041 -0.11632041 ... -0.17557865 -0.17557865\n",
      "  -0.17557865]\n",
      " ...\n",
      " [-0.13093327 -0.13093327 -0.13093327 ... -0.14237332 -0.14237332\n",
      "  -0.14237332]\n",
      " [-0.1271075  -0.1271075  -0.1271075  ... -0.16766535 -0.16766535\n",
      "  -0.16766535]\n",
      " [-0.16502622 -0.16502622 -0.16502622 ... -0.18193355 -0.18193355\n",
      "  -0.18193355]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.14253503 -0.14253503 -0.14253503 ... -0.16362204 -0.16362204\n",
      "  -0.16362204]\n",
      " [-0.16887572 -0.16887572 -0.16887572 ... -0.17062588 -0.17062588\n",
      "  -0.17062588]\n",
      " [-0.16398257 -0.16398257 -0.16398257 ... -0.2138996  -0.2138996\n",
      "  -0.2138996 ]\n",
      " ...\n",
      " [-0.13861682 -0.13861682 -0.13861682 ... -0.17234063 -0.17234063\n",
      "  -0.17234063]\n",
      " [-0.18770203 -0.18770203 -0.18770203 ... -0.13176277 -0.13176277\n",
      "  -0.13176277]\n",
      " [-0.15724567 -0.15724567 -0.15724567 ... -0.15199097 -0.15199097\n",
      "  -0.15199097]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18079367 -0.18079367 -0.18079367 ... -0.16264224 -0.16264224\n",
      "  -0.16264224]\n",
      " [-0.11557864 -0.11557864 -0.11557864 ... -0.19895393 -0.19895393\n",
      "  -0.19895393]\n",
      " [-0.116504   -0.116504   -0.116504   ... -0.1617563  -0.1617563\n",
      "  -0.1617563 ]\n",
      " ...\n",
      " [-0.1659827  -0.1659827  -0.1659827  ... -0.14978272 -0.14978272\n",
      "  -0.14978272]\n",
      " [-0.17022364 -0.17022364 -0.17022364 ... -0.13258436 -0.13258436\n",
      "  -0.13258436]\n",
      " [-0.13540906 -0.13540906 -0.13540906 ... -0.11417179 -0.11417179\n",
      "  -0.11417179]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.13784073 -0.13784073 -0.13784073 ... -0.10328784 -0.10328784\n",
      "  -0.10328784]\n",
      " [-0.16422957 -0.16422957 -0.16422957 ... -0.11461127 -0.11461127\n",
      "  -0.11461127]\n",
      " [-0.14171802 -0.14171802 -0.14171802 ... -0.1296028  -0.1296028\n",
      "  -0.1296028 ]\n",
      " ...\n",
      " [-0.1792196  -0.1792196  -0.1792196  ... -0.13088304 -0.13088304\n",
      "  -0.13088304]\n",
      " [-0.17063235 -0.17063235 -0.17063235 ... -0.1398833  -0.1398833\n",
      "  -0.1398833 ]\n",
      " [-0.17061673 -0.17061673 -0.17061673 ... -0.15871863 -0.15871863\n",
      "  -0.15871863]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   3939  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.1648492  -0.1648492  -0.1648492  ... -0.20128278 -0.20128278\n",
      "  -0.20128278]\n",
      " [-0.16366127 -0.16366127 -0.16366127 ... -0.1315224  -0.1315224\n",
      "  -0.1315224 ]\n",
      " [-0.15200703 -0.15200703 -0.15200703 ... -0.1441597  -0.1441597\n",
      "  -0.1441597 ]\n",
      " ...\n",
      " [-0.14333926 -0.14333926 -0.14333926 ... -0.12945291 -0.12945291\n",
      "  -0.12945291]\n",
      " [-0.16751003 -0.16751003 -0.16751003 ... -0.17189975 -0.17189975\n",
      "  -0.17189975]\n",
      " [-0.17307985 -0.17307985 -0.17307985 ... -0.18550155 -0.18550155\n",
      "  -0.18550155]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.13603377 -0.13603377 -0.13603377 ... -0.14193928 -0.14193928\n",
      "  -0.14193928]\n",
      " [-0.17302898 -0.17302898 -0.17302898 ... -0.16527179 -0.16527179\n",
      "  -0.16527179]\n",
      " [-0.15829796 -0.15829796 -0.15829796 ... -0.19514327 -0.19514327\n",
      "  -0.19514327]\n",
      " ...\n",
      " [-0.18356614 -0.18356614 -0.18356614 ... -0.12767595 -0.12767595\n",
      "  -0.12767595]\n",
      " [-0.13995893 -0.13995893 -0.13995893 ... -0.1610304  -0.1610304\n",
      "  -0.1610304 ]\n",
      " [-0.1772232  -0.1772232  -0.1772232  ... -0.16510725 -0.16510725\n",
      "  -0.16510725]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13998716 -0.13998716 -0.13998716 ... -0.16623402 -0.16623402\n",
      "  -0.16623402]\n",
      " [-0.2005308  -0.2005308  -0.2005308  ... -0.18891172 -0.18891172\n",
      "  -0.18891172]\n",
      " [-0.16274181 -0.16274181 -0.16274181 ... -0.17814922 -0.17814922\n",
      "  -0.17814922]\n",
      " ...\n",
      " [-0.12115128 -0.12115128 -0.12115128 ... -0.16494963 -0.16494963\n",
      "  -0.16494963]\n",
      " [-0.17609382 -0.17609382 -0.17609382 ... -0.16365707 -0.16365707\n",
      "  -0.16365707]\n",
      " [-0.21512814 -0.21512814 -0.21512814 ... -0.16589154 -0.16589154\n",
      "  -0.16589154]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18916008 -0.18916008 -0.18916008 ... -0.10448869 -0.10448869\n",
      "  -0.10448869]\n",
      " [-0.17965724 -0.17965724 -0.17965724 ... -0.193976   -0.193976\n",
      "  -0.193976  ]\n",
      " [-0.18024188 -0.18024188 -0.18024188 ... -0.18940318 -0.18940318\n",
      "  -0.18940318]\n",
      " ...\n",
      " [-0.15717036 -0.15717036 -0.15717036 ... -0.16093814 -0.16093814\n",
      "  -0.16093814]\n",
      " [-0.18452677 -0.18452677 -0.18452677 ... -0.15871586 -0.15871586\n",
      "  -0.15871586]\n",
      " [-0.16269355 -0.16269355 -0.16269355 ... -0.18495557 -0.18495557\n",
      "  -0.18495557]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.11478876 -0.11478876 -0.11478876 ... -0.17895465 -0.17895465\n",
      "  -0.17895465]\n",
      " [-0.16863704 -0.16863704 -0.16863704 ... -0.15202905 -0.15202905\n",
      "  -0.15202905]\n",
      " [-0.11587879 -0.11587879 -0.11587879 ... -0.17578784 -0.17578784\n",
      "  -0.17578784]\n",
      " ...\n",
      " [-0.21166411 -0.21166411 -0.21166411 ... -0.16205391 -0.16205391\n",
      "  -0.16205391]\n",
      " [-0.1711608  -0.1711608  -0.1711608  ... -0.14805758 -0.14805758\n",
      "  -0.14805758]\n",
      " [-0.17566553 -0.17566553 -0.17566553 ... -0.15391651 -0.15391651\n",
      "  -0.15391651]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1741364  -0.1741364  -0.1741364  ... -0.1318368  -0.1318368\n",
      "  -0.1318368 ]\n",
      " [-0.15460293 -0.15460293 -0.15460293 ... -0.18068251 -0.18068251\n",
      "  -0.18068251]\n",
      " [-0.12470427 -0.12470427 -0.12470427 ... -0.11893053 -0.11893053\n",
      "  -0.11893053]\n",
      " ...\n",
      " [-0.15815023 -0.15815023 -0.15815023 ... -0.17465867 -0.17465867\n",
      "  -0.17465867]\n",
      " [-0.15231906 -0.15231906 -0.15231906 ... -0.13930033 -0.13930033\n",
      "  -0.13930033]\n",
      " [-0.12805429 -0.12805429 -0.12805429 ... -0.1554603  -0.1554603\n",
      "  -0.1554603 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.1581639  -0.1581639  -0.1581639  ... -0.1649494  -0.1649494\n",
      "  -0.1649494 ]\n",
      " [-0.16531043 -0.16531043 -0.16531043 ... -0.1623956  -0.1623956\n",
      "  -0.1623956 ]\n",
      " [-0.2040658  -0.2040658  -0.2040658  ... -0.17488202 -0.17488202\n",
      "  -0.17488202]\n",
      " ...\n",
      " [-0.19753109 -0.19753109 -0.19753109 ... -0.17136727 -0.17136727\n",
      "  -0.17136727]\n",
      " [-0.16772254 -0.16772254 -0.16772254 ... -0.18079363 -0.18079363\n",
      "  -0.18079363]\n",
      " [-0.19798356 -0.19798356 -0.19798356 ... -0.17095943 -0.17095943\n",
      "  -0.17095943]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.1530214  -0.1530214  -0.1530214  ... -0.13024205 -0.13024205\n",
      "  -0.13024205]\n",
      " [-0.17781746 -0.17781746 -0.17781746 ... -0.1147317  -0.1147317\n",
      "  -0.1147317 ]\n",
      " [-0.12895627 -0.12895627 -0.12895627 ... -0.15675172 -0.15675172\n",
      "  -0.15675172]\n",
      " ...\n",
      " [-0.20642678 -0.20642678 -0.20642678 ... -0.16894804 -0.16894804\n",
      "  -0.16894804]\n",
      " [-0.17917608 -0.17917608 -0.17917608 ... -0.1814112  -0.1814112\n",
      "  -0.1814112 ]\n",
      " [-0.15830801 -0.15830801 -0.15830801 ... -0.15295804 -0.15295804\n",
      "  -0.15295804]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "------------------------------------------------------------  step step   4040  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.18478659 -0.18478659 -0.18478659 ... -0.1899313  -0.1899313\n",
      "  -0.1899313 ]\n",
      " [-0.17649323 -0.17649323 -0.17649323 ... -0.17079517 -0.17079517\n",
      "  -0.17079517]\n",
      " [-0.18804257 -0.18804257 -0.18804257 ... -0.15337448 -0.15337448\n",
      "  -0.15337448]\n",
      " ...\n",
      " [-0.17196763 -0.17196763 -0.17196763 ... -0.17711207 -0.17711207\n",
      "  -0.17711207]\n",
      " [-0.1856257  -0.1856257  -0.1856257  ... -0.21597902 -0.21597902\n",
      "  -0.21597902]\n",
      " [-0.19354095 -0.19354095 -0.19354095 ... -0.19190468 -0.19190468\n",
      "  -0.19190468]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22764747 -0.22764747 -0.22764747 ... -0.18639454 -0.18639454\n",
      "  -0.18639454]\n",
      " [-0.1549012  -0.1549012  -0.1549012  ... -0.14557856 -0.14557856\n",
      "  -0.14557856]\n",
      " [-0.19227196 -0.19227196 -0.19227196 ... -0.17354462 -0.17354462\n",
      "  -0.17354462]\n",
      " ...\n",
      " [-0.18484977 -0.18484977 -0.18484977 ... -0.19804016 -0.19804016\n",
      "  -0.19804016]\n",
      " [-0.19054805 -0.19054805 -0.19054805 ... -0.15773596 -0.15773596\n",
      "  -0.15773596]\n",
      " [-0.16633312 -0.16633312 -0.16633312 ... -0.1884637  -0.1884637\n",
      "  -0.1884637 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.13369608 -0.13369608 -0.13369608 ... -0.18920939 -0.18920939\n",
      "  -0.18920939]\n",
      " [-0.14682654 -0.14682654 -0.14682654 ... -0.17795296 -0.17795296\n",
      "  -0.17795296]\n",
      " [-0.15913701 -0.15913701 -0.15913701 ... -0.19565755 -0.19565755\n",
      "  -0.19565755]\n",
      " ...\n",
      " [-0.17512968 -0.17512968 -0.17512968 ... -0.15022938 -0.15022938\n",
      "  -0.15022938]\n",
      " [-0.18026417 -0.18026417 -0.18026417 ... -0.18524441 -0.18524441\n",
      "  -0.18524441]\n",
      " [-0.15372533 -0.15372533 -0.15372533 ... -0.15541562 -0.15541562\n",
      "  -0.15541562]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.17421421 -0.17421421 -0.17421421 ... -0.17424616 -0.17424616\n",
      "  -0.17424616]\n",
      " [-0.2087037  -0.2087037  -0.2087037  ... -0.14959446 -0.14959446\n",
      "  -0.14959446]\n",
      " [-0.18898197 -0.18898197 -0.18898197 ... -0.18162854 -0.18162854\n",
      "  -0.18162854]\n",
      " ...\n",
      " [-0.14441219 -0.14441219 -0.14441219 ... -0.17868905 -0.17868905\n",
      "  -0.17868905]\n",
      " [-0.14092016 -0.14092016 -0.14092016 ... -0.16117501 -0.16117501\n",
      "  -0.16117501]\n",
      " [-0.19448498 -0.19448498 -0.19448498 ... -0.19224347 -0.19224347\n",
      "  -0.19224347]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.14670214 -0.14670214 -0.14670214 ... -0.1523164  -0.1523164\n",
      "  -0.1523164 ]\n",
      " [-0.18418339 -0.18418339 -0.18418339 ... -0.16919132 -0.16919132\n",
      "  -0.16919132]\n",
      " [-0.1578069  -0.1578069  -0.1578069  ... -0.1820644  -0.1820644\n",
      "  -0.1820644 ]\n",
      " ...\n",
      " [-0.14243269 -0.14243269 -0.14243269 ... -0.17708308 -0.17708308\n",
      "  -0.17708308]\n",
      " [-0.17807823 -0.17807823 -0.17807823 ... -0.1514872  -0.1514872\n",
      "  -0.1514872 ]\n",
      " [-0.16545098 -0.16545098 -0.16545098 ... -0.19027108 -0.19027108\n",
      "  -0.19027108]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.23063153 -0.23063153 -0.23063153 ... -0.14859116 -0.14859116\n",
      "  -0.14859116]\n",
      " [-0.17656955 -0.17656955 -0.17656955 ... -0.1687172  -0.1687172\n",
      "  -0.1687172 ]\n",
      " [-0.17838474 -0.17838474 -0.17838474 ... -0.19209486 -0.19209486\n",
      "  -0.19209486]\n",
      " ...\n",
      " [-0.15956108 -0.15956108 -0.15956108 ... -0.16459085 -0.16459085\n",
      "  -0.16459085]\n",
      " [-0.17470922 -0.17470922 -0.17470922 ... -0.187361   -0.187361\n",
      "  -0.187361  ]\n",
      " [-0.18290927 -0.18290927 -0.18290927 ... -0.1388298  -0.1388298\n",
      "  -0.1388298 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2054398  -0.2054398  -0.2054398  ... -0.18753849 -0.18753849\n",
      "  -0.18753849]\n",
      " [-0.18619365 -0.18619365 -0.18619365 ... -0.16643704 -0.16643704\n",
      "  -0.16643704]\n",
      " [-0.20773055 -0.20773055 -0.20773055 ... -0.18829525 -0.18829525\n",
      "  -0.18829525]\n",
      " ...\n",
      " [-0.17751488 -0.17751488 -0.17751488 ... -0.1829401  -0.1829401\n",
      "  -0.1829401 ]\n",
      " [-0.14812344 -0.14812344 -0.14812344 ... -0.17178506 -0.17178506\n",
      "  -0.17178506]\n",
      " [-0.18392216 -0.18392216 -0.18392216 ... -0.15271497 -0.15271497\n",
      "  -0.15271497]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15685914 -0.15685914 -0.15685914 ... -0.20156023 -0.20156023\n",
      "  -0.20156023]\n",
      " [-0.1738337  -0.1738337  -0.1738337  ... -0.18462738 -0.18462738\n",
      "  -0.18462738]\n",
      " [-0.14100543 -0.14100543 -0.14100543 ... -0.1912844  -0.1912844\n",
      "  -0.1912844 ]\n",
      " ...\n",
      " [-0.16574734 -0.16574734 -0.16574734 ... -0.16688998 -0.16688998\n",
      "  -0.16688998]\n",
      " [-0.16588551 -0.16588551 -0.16588551 ... -0.19077927 -0.19077927\n",
      "  -0.19077927]\n",
      " [-0.09582164 -0.09582164 -0.09582164 ... -0.15209037 -0.15209037\n",
      "  -0.15209037]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4141  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20233767 -0.20233767 -0.20233767 ... -0.20263219 -0.20263219\n",
      "  -0.20263219]\n",
      " [-0.21458007 -0.21458007 -0.21458007 ... -0.20682746 -0.20682746\n",
      "  -0.20682746]\n",
      " [-0.16927955 -0.16927955 -0.16927955 ... -0.2265161  -0.2265161\n",
      "  -0.2265161 ]\n",
      " ...\n",
      " [-0.18254454 -0.18254454 -0.18254454 ... -0.21951208 -0.21951208\n",
      "  -0.21951208]\n",
      " [-0.18452258 -0.18452258 -0.18452258 ... -0.23651193 -0.23651193\n",
      "  -0.23651193]\n",
      " [-0.20070302 -0.20070302 -0.20070302 ... -0.18111359 -0.18111359\n",
      "  -0.18111359]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.19738786 -0.19738786 -0.19738786 ... -0.18061432 -0.18061432\n",
      "  -0.18061432]\n",
      " [-0.21058187 -0.21058187 -0.21058187 ... -0.19106345 -0.19106345\n",
      "  -0.19106345]\n",
      " [-0.21533021 -0.21533021 -0.21533021 ... -0.18554834 -0.18554834\n",
      "  -0.18554834]\n",
      " ...\n",
      " [-0.22183451 -0.22183451 -0.22183451 ... -0.21166766 -0.21166766\n",
      "  -0.21166766]\n",
      " [-0.19610399 -0.19610399 -0.19610399 ... -0.20740686 -0.20740686\n",
      "  -0.20740686]\n",
      " [-0.22998948 -0.22998948 -0.22998948 ... -0.21447016 -0.21447016\n",
      "  -0.21447016]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.17801672 -0.17801672 -0.17801672 ... -0.174116   -0.174116\n",
      "  -0.174116  ]\n",
      " [-0.20540845 -0.20540845 -0.20540845 ... -0.18327022 -0.18327022\n",
      "  -0.18327022]\n",
      " [-0.20566952 -0.20566952 -0.20566952 ... -0.17707571 -0.17707571\n",
      "  -0.17707571]\n",
      " ...\n",
      " [-0.19903332 -0.19903332 -0.19903332 ... -0.20318444 -0.20318444\n",
      "  -0.20318444]\n",
      " [-0.16248672 -0.16248672 -0.16248672 ... -0.16062589 -0.16062589\n",
      "  -0.16062589]\n",
      " [-0.21141851 -0.21141851 -0.21141851 ... -0.17324366 -0.17324366\n",
      "  -0.17324366]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.18412417 -0.18412417 -0.18412417 ... -0.1665053  -0.1665053\n",
      "  -0.1665053 ]\n",
      " [-0.16631612 -0.16631612 -0.16631612 ... -0.20207255 -0.20207255\n",
      "  -0.20207255]\n",
      " [-0.24773963 -0.24773963 -0.24773963 ... -0.20041755 -0.20041755\n",
      "  -0.20041755]\n",
      " ...\n",
      " [-0.1548082  -0.1548082  -0.1548082  ... -0.18819608 -0.18819608\n",
      "  -0.18819608]\n",
      " [-0.19790384 -0.19790384 -0.19790384 ... -0.17428595 -0.17428595\n",
      "  -0.17428595]\n",
      " [-0.21339805 -0.21339805 -0.21339805 ... -0.17877059 -0.17877059\n",
      "  -0.17877059]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.18790597 -0.18790597 -0.18790597 ... -0.16746643 -0.16746643\n",
      "  -0.16746643]\n",
      " [-0.1997064  -0.1997064  -0.1997064  ... -0.24125175 -0.24125175\n",
      "  -0.24125175]\n",
      " [-0.19436827 -0.19436827 -0.19436827 ... -0.18232018 -0.18232018\n",
      "  -0.18232018]\n",
      " ...\n",
      " [-0.19500598 -0.19500598 -0.19500598 ... -0.15189677 -0.15189677\n",
      "  -0.15189677]\n",
      " [-0.23239473 -0.23239473 -0.23239473 ... -0.1946903  -0.1946903\n",
      "  -0.1946903 ]\n",
      " [-0.16455835 -0.16455835 -0.16455835 ... -0.18653572 -0.18653572\n",
      "  -0.18653572]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.17046535 -0.17046535 -0.17046535 ... -0.20105743 -0.20105743\n",
      "  -0.20105743]\n",
      " [-0.19397444 -0.19397444 -0.19397444 ... -0.16295865 -0.16295865\n",
      "  -0.16295865]\n",
      " [-0.18168822 -0.18168822 -0.18168822 ... -0.18963909 -0.18963909\n",
      "  -0.18963909]\n",
      " ...\n",
      " [-0.19682522 -0.19682522 -0.19682522 ... -0.2131353  -0.2131353\n",
      "  -0.2131353 ]\n",
      " [-0.16878363 -0.16878363 -0.16878363 ... -0.17417218 -0.17417218\n",
      "  -0.17417218]\n",
      " [-0.17172375 -0.17172375 -0.17172375 ... -0.18355277 -0.18355277\n",
      "  -0.18355277]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18946812 -0.18946812 -0.18946812 ... -0.20699811 -0.20699811\n",
      "  -0.20699811]\n",
      " [-0.19358906 -0.19358906 -0.19358906 ... -0.18495737 -0.18495737\n",
      "  -0.18495737]\n",
      " [-0.14961721 -0.14961721 -0.14961721 ... -0.18300867 -0.18300867\n",
      "  -0.18300867]\n",
      " ...\n",
      " [-0.22065945 -0.22065945 -0.22065945 ... -0.15639988 -0.15639988\n",
      "  -0.15639988]\n",
      " [-0.24640734 -0.24640734 -0.24640734 ... -0.20252807 -0.20252807\n",
      "  -0.20252807]\n",
      " [-0.18904531 -0.18904531 -0.18904531 ... -0.19768229 -0.19768229\n",
      "  -0.19768229]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.20220831 -0.20220831 -0.20220831 ... -0.16949435 -0.16949435\n",
      "  -0.16949435]\n",
      " [-0.17257765 -0.17257765 -0.17257765 ... -0.16209388 -0.16209388\n",
      "  -0.16209388]\n",
      " [-0.1127792  -0.1127792  -0.1127792  ... -0.16616365 -0.16616365\n",
      "  -0.16616365]\n",
      " ...\n",
      " [-0.19795749 -0.19795749 -0.19795749 ... -0.19234727 -0.19234727\n",
      "  -0.19234727]\n",
      " [-0.18919998 -0.18919998 -0.18919998 ... -0.23314708 -0.23314708\n",
      "  -0.23314708]\n",
      " [-0.16523801 -0.16523801 -0.16523801 ... -0.19004163 -0.19004163\n",
      "  -0.19004163]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4242  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.22078344 -0.22078344 -0.22078344 ... -0.18949676 -0.18949676\n",
      "  -0.18949676]\n",
      " [-0.23365921 -0.23365921 -0.23365921 ... -0.22140697 -0.22140697\n",
      "  -0.22140697]\n",
      " [-0.21895036 -0.21895036 -0.21895036 ... -0.19156113 -0.19156113\n",
      "  -0.19156113]\n",
      " ...\n",
      " [-0.22157115 -0.22157115 -0.22157115 ... -0.14559269 -0.14559269\n",
      "  -0.14559269]\n",
      " [-0.17978376 -0.17978376 -0.17978376 ... -0.14467242 -0.14467242\n",
      "  -0.14467242]\n",
      " [-0.19756514 -0.19756514 -0.19756514 ... -0.17799252 -0.17799252\n",
      "  -0.17799252]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.18293741 -0.18293741 -0.18293741 ... -0.21235327 -0.21235327\n",
      "  -0.21235327]\n",
      " [-0.17226887 -0.17226887 -0.17226887 ... -0.18585601 -0.18585601\n",
      "  -0.18585601]\n",
      " [-0.23628515 -0.23628515 -0.23628515 ... -0.16069217 -0.16069217\n",
      "  -0.16069217]\n",
      " ...\n",
      " [-0.22082846 -0.22082846 -0.22082846 ... -0.17208692 -0.17208692\n",
      "  -0.17208692]\n",
      " [-0.20034735 -0.20034735 -0.20034735 ... -0.19174546 -0.19174546\n",
      "  -0.19174546]\n",
      " [-0.18425836 -0.18425836 -0.18425836 ... -0.1562178  -0.1562178\n",
      "  -0.1562178 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18907301 -0.18907301 -0.18907301 ... -0.14661287 -0.14661287\n",
      "  -0.14661287]\n",
      " [-0.1639885  -0.1639885  -0.1639885  ... -0.19698308 -0.19698308\n",
      "  -0.19698308]\n",
      " [-0.24205083 -0.24205083 -0.24205083 ... -0.18934952 -0.18934952\n",
      "  -0.18934952]\n",
      " ...\n",
      " [-0.16473532 -0.16473532 -0.16473532 ... -0.20108144 -0.20108144\n",
      "  -0.20108144]\n",
      " [-0.17650233 -0.17650233 -0.17650233 ... -0.16142012 -0.16142012\n",
      "  -0.16142012]\n",
      " [-0.2229692  -0.2229692  -0.2229692  ... -0.17463224 -0.17463224\n",
      "  -0.17463224]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.15780716 -0.15780716 -0.15780716 ... -0.18167895 -0.18167895\n",
      "  -0.18167895]\n",
      " [-0.20024806 -0.20024806 -0.20024806 ... -0.16501977 -0.16501977\n",
      "  -0.16501977]\n",
      " [-0.14385927 -0.14385927 -0.14385927 ... -0.16985458 -0.16985458\n",
      "  -0.16985458]\n",
      " ...\n",
      " [-0.16549549 -0.16549549 -0.16549549 ... -0.2099069  -0.2099069\n",
      "  -0.2099069 ]\n",
      " [-0.196811   -0.196811   -0.196811   ... -0.21486814 -0.21486814\n",
      "  -0.21486814]\n",
      " [-0.23533417 -0.23533417 -0.23533417 ... -0.19979069 -0.19979069\n",
      "  -0.19979069]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.21465573 -0.21465573 -0.21465573 ... -0.22436312 -0.22436312\n",
      "  -0.22436312]\n",
      " [-0.22589427 -0.22589427 -0.22589427 ... -0.21576276 -0.21576276\n",
      "  -0.21576276]\n",
      " [-0.19167176 -0.19167176 -0.19167176 ... -0.14702143 -0.14702143\n",
      "  -0.14702143]\n",
      " ...\n",
      " [-0.19376385 -0.19376385 -0.19376385 ... -0.17944437 -0.17944437\n",
      "  -0.17944437]\n",
      " [-0.17556955 -0.17556955 -0.17556955 ... -0.1653204  -0.1653204\n",
      "  -0.1653204 ]\n",
      " [-0.20857863 -0.20857863 -0.20857863 ... -0.20476031 -0.20476031\n",
      "  -0.20476031]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.16983932 -0.16983932 -0.16983932 ... -0.20705335 -0.20705335\n",
      "  -0.20705335]\n",
      " [-0.2066825  -0.2066825  -0.2066825  ... -0.18662593 -0.18662593\n",
      "  -0.18662593]\n",
      " [-0.22156033 -0.22156033 -0.22156033 ... -0.18917611 -0.18917611\n",
      "  -0.18917611]\n",
      " ...\n",
      " [-0.20086372 -0.20086372 -0.20086372 ... -0.21794896 -0.21794896\n",
      "  -0.21794896]\n",
      " [-0.20554101 -0.20554101 -0.20554101 ... -0.22674328 -0.22674328\n",
      "  -0.22674328]\n",
      " [-0.16745889 -0.16745889 -0.16745889 ... -0.21195745 -0.21195745\n",
      "  -0.21195745]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2147873  -0.2147873  -0.2147873  ... -0.22993135 -0.22993135\n",
      "  -0.22993135]\n",
      " [-0.1279121  -0.1279121  -0.1279121  ... -0.21897954 -0.21897954\n",
      "  -0.21897954]\n",
      " [-0.24928756 -0.24928756 -0.24928756 ... -0.19429623 -0.19429623\n",
      "  -0.19429623]\n",
      " ...\n",
      " [-0.18440723 -0.18440723 -0.18440723 ... -0.1812016  -0.1812016\n",
      "  -0.1812016 ]\n",
      " [-0.20713553 -0.20713553 -0.20713553 ... -0.15873127 -0.15873127\n",
      "  -0.15873127]\n",
      " [-0.16520615 -0.16520615 -0.16520615 ... -0.16016732 -0.16016732\n",
      "  -0.16016732]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.17636444 -0.17636444 -0.17636444 ... -0.18825069 -0.18825069\n",
      "  -0.18825069]\n",
      " [-0.11562872 -0.11562872 -0.11562872 ... -0.19197507 -0.19197507\n",
      "  -0.19197507]\n",
      " [-0.22103913 -0.22103913 -0.22103913 ... -0.22301686 -0.22301686\n",
      "  -0.22301686]\n",
      " ...\n",
      " [-0.19114287 -0.19114287 -0.19114287 ... -0.21150169 -0.21150169\n",
      "  -0.21150169]\n",
      " [-0.21160972 -0.21160972 -0.21160972 ... -0.17058182 -0.17058182\n",
      "  -0.17058182]\n",
      " [-0.2138911  -0.2138911  -0.2138911  ... -0.18320262 -0.18320262\n",
      "  -0.18320262]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4343  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.16903637 -0.16903637 -0.16903637 ... -0.18917917 -0.18917917\n",
      "  -0.18917917]\n",
      " [-0.20045108 -0.20045108 -0.20045108 ... -0.21785264 -0.21785264\n",
      "  -0.21785264]\n",
      " [-0.21391103 -0.21391103 -0.21391103 ... -0.17889702 -0.17889702\n",
      "  -0.17889702]\n",
      " ...\n",
      " [-0.21389633 -0.21389633 -0.21389633 ... -0.20312464 -0.20312464\n",
      "  -0.20312464]\n",
      " [-0.21414283 -0.21414283 -0.21414283 ... -0.24248484 -0.24248484\n",
      "  -0.24248484]\n",
      " [-0.18587068 -0.18587068 -0.18587068 ... -0.19247335 -0.19247335\n",
      "  -0.19247335]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.196242   -0.196242   -0.196242   ... -0.19307607 -0.19307607\n",
      "  -0.19307607]\n",
      " [-0.17551212 -0.17551212 -0.17551212 ... -0.23728892 -0.23728892\n",
      "  -0.23728892]\n",
      " [-0.24437374 -0.24437374 -0.24437374 ... -0.24552672 -0.24552672\n",
      "  -0.24552672]\n",
      " ...\n",
      " [-0.20181502 -0.20181502 -0.20181502 ... -0.19036154 -0.19036154\n",
      "  -0.19036154]\n",
      " [-0.1765454  -0.1765454  -0.1765454  ... -0.22515853 -0.22515853\n",
      "  -0.22515853]\n",
      " [-0.1660667  -0.1660667  -0.1660667  ... -0.23789641 -0.23789641\n",
      "  -0.23789641]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18971157 -0.18971157 -0.18971157 ... -0.2147985  -0.2147985\n",
      "  -0.2147985 ]\n",
      " [-0.24047932 -0.24047932 -0.24047932 ... -0.22647296 -0.22647296\n",
      "  -0.22647296]\n",
      " [-0.19861668 -0.19861668 -0.19861668 ... -0.12302334 -0.12302334\n",
      "  -0.12302334]\n",
      " ...\n",
      " [-0.19601071 -0.19601071 -0.19601071 ... -0.20701039 -0.20701039\n",
      "  -0.20701039]\n",
      " [-0.23146096 -0.23146096 -0.23146096 ... -0.21801332 -0.21801332\n",
      "  -0.21801332]\n",
      " [-0.17868681 -0.17868681 -0.17868681 ... -0.19648272 -0.19648272\n",
      "  -0.19648272]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2441226  -0.2441226  -0.2441226  ... -0.1991542  -0.1991542\n",
      "  -0.1991542 ]\n",
      " [-0.19264387 -0.19264387 -0.19264387 ... -0.2505738  -0.2505738\n",
      "  -0.2505738 ]\n",
      " [-0.20595089 -0.20595089 -0.20595089 ... -0.20715776 -0.20715776\n",
      "  -0.20715776]\n",
      " ...\n",
      " [-0.17719418 -0.17719418 -0.17719418 ... -0.19295345 -0.19295345\n",
      "  -0.19295345]\n",
      " [-0.20806818 -0.20806818 -0.20806818 ... -0.18813981 -0.18813981\n",
      "  -0.18813981]\n",
      " [-0.21610966 -0.21610966 -0.21610966 ... -0.19112007 -0.19112007\n",
      "  -0.19112007]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.21762636 -0.21762636 -0.21762636 ... -0.2175397  -0.2175397\n",
      "  -0.2175397 ]\n",
      " [-0.19721231 -0.19721231 -0.19721231 ... -0.20997284 -0.20997284\n",
      "  -0.20997284]\n",
      " [-0.18108317 -0.18108317 -0.18108317 ... -0.17802337 -0.17802337\n",
      "  -0.17802337]\n",
      " ...\n",
      " [-0.21762636 -0.21762636 -0.21762636 ... -0.21577966 -0.21577966\n",
      "  -0.21577966]\n",
      " [-0.24694891 -0.24694891 -0.24694891 ... -0.22162324 -0.22162324\n",
      "  -0.22162324]\n",
      " [-0.1843204  -0.1843204  -0.1843204  ... -0.23220924 -0.23220924\n",
      "  -0.23220924]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.2012202  -0.2012202  -0.2012202  ... -0.1855292  -0.1855292\n",
      "  -0.1855292 ]\n",
      " [-0.22771573 -0.22771573 -0.22771573 ... -0.22566986 -0.22566986\n",
      "  -0.22566986]\n",
      " [-0.19910006 -0.19910006 -0.19910006 ... -0.18962704 -0.18962704\n",
      "  -0.18962704]\n",
      " ...\n",
      " [-0.22525227 -0.22525227 -0.22525227 ... -0.2036439  -0.2036439\n",
      "  -0.2036439 ]\n",
      " [-0.16760488 -0.16760488 -0.16760488 ... -0.19809149 -0.19809149\n",
      "  -0.19809149]\n",
      " [-0.20714298 -0.20714298 -0.20714298 ... -0.20079021 -0.20079021\n",
      "  -0.20079021]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21499215 -0.21499215 -0.21499215 ... -0.20950267 -0.20950267\n",
      "  -0.20950267]\n",
      " [-0.2259718  -0.2259718  -0.2259718  ... -0.18700492 -0.18700492\n",
      "  -0.18700492]\n",
      " [-0.22243045 -0.22243045 -0.22243045 ... -0.19836967 -0.19836967\n",
      "  -0.19836967]\n",
      " ...\n",
      " [-0.1815438  -0.1815438  -0.1815438  ... -0.21649483 -0.21649483\n",
      "  -0.21649483]\n",
      " [-0.2063301  -0.2063301  -0.2063301  ... -0.22546908 -0.22546908\n",
      "  -0.22546908]\n",
      " [-0.21445927 -0.21445927 -0.21445927 ... -0.23695949 -0.23695949\n",
      "  -0.23695949]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.16776258 -0.16776258 -0.16776258 ... -0.1890461  -0.1890461\n",
      "  -0.1890461 ]\n",
      " [-0.22163822 -0.22163822 -0.22163822 ... -0.23547976 -0.23547976\n",
      "  -0.23547976]\n",
      " [-0.1876063  -0.1876063  -0.1876063  ... -0.19503662 -0.19503662\n",
      "  -0.19503662]\n",
      " ...\n",
      " [-0.21888861 -0.21888861 -0.21888861 ... -0.21139279 -0.21139279\n",
      "  -0.21139279]\n",
      " [-0.25213265 -0.25213265 -0.25213265 ... -0.20667863 -0.20667863\n",
      "  -0.20667863]\n",
      " [-0.20567194 -0.20567194 -0.20567194 ... -0.21531504 -0.21531504\n",
      "  -0.21531504]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4444  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23230612 -0.23230612 -0.23230612 ... -0.23454793 -0.23454793\n",
      "  -0.23454793]\n",
      " [-0.24318242 -0.24318242 -0.24318242 ... -0.23873319 -0.23873319\n",
      "  -0.23873319]\n",
      " [-0.22219    -0.22219    -0.22219    ... -0.21932155 -0.21932155\n",
      "  -0.21932155]\n",
      " ...\n",
      " [-0.23397337 -0.23397337 -0.23397337 ... -0.19928938 -0.19928938\n",
      "  -0.19928938]\n",
      " [-0.21922664 -0.21922664 -0.21922664 ... -0.21293041 -0.21293041\n",
      "  -0.21293041]\n",
      " [-0.23530929 -0.23530929 -0.23530929 ... -0.24623074 -0.24623074\n",
      "  -0.24623074]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.1969389  -0.1969389  -0.1969389  ... -0.18831715 -0.18831715\n",
      "  -0.18831715]\n",
      " [-0.23472244 -0.23472244 -0.23472244 ... -0.18441144 -0.18441144\n",
      "  -0.18441144]\n",
      " [-0.21391138 -0.21391138 -0.21391138 ... -0.2573406  -0.2573406\n",
      "  -0.2573406 ]\n",
      " ...\n",
      " [-0.24211448 -0.24211448 -0.24211448 ... -0.23093644 -0.23093644\n",
      "  -0.23093644]\n",
      " [-0.20935616 -0.20935616 -0.20935616 ... -0.21533462 -0.21533462\n",
      "  -0.21533462]\n",
      " [-0.20840207 -0.20840207 -0.20840207 ... -0.21350688 -0.21350688\n",
      "  -0.21350688]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23428556 -0.23428556 -0.23428556 ... -0.22917897 -0.22917897\n",
      "  -0.22917897]\n",
      " [-0.24710628 -0.24710628 -0.24710628 ... -0.18398422 -0.18398422\n",
      "  -0.18398422]\n",
      " [-0.1818767  -0.1818767  -0.1818767  ... -0.2252244  -0.2252244\n",
      "  -0.2252244 ]\n",
      " ...\n",
      " [-0.2041134  -0.2041134  -0.2041134  ... -0.2288698  -0.2288698\n",
      "  -0.2288698 ]\n",
      " [-0.21506053 -0.21506053 -0.21506053 ... -0.2126172  -0.2126172\n",
      "  -0.2126172 ]\n",
      " [-0.24516428 -0.24516428 -0.24516428 ... -0.2074202  -0.2074202\n",
      "  -0.2074202 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24772167 -0.24772167 -0.24772167 ... -0.23074794 -0.23074794\n",
      "  -0.23074794]\n",
      " [-0.23763818 -0.23763818 -0.23763818 ... -0.2190885  -0.2190885\n",
      "  -0.2190885 ]\n",
      " [-0.19921964 -0.19921964 -0.19921964 ... -0.23696321 -0.23696321\n",
      "  -0.23696321]\n",
      " ...\n",
      " [-0.26927358 -0.26927358 -0.26927358 ... -0.22000188 -0.22000188\n",
      "  -0.22000188]\n",
      " [-0.24632043 -0.24632043 -0.24632043 ... -0.22634178 -0.22634178\n",
      "  -0.22634178]\n",
      " [-0.22405845 -0.22405845 -0.22405845 ... -0.1985451  -0.1985451\n",
      "  -0.1985451 ]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23361634 -0.23361634 -0.23361634 ... -0.22457527 -0.22457527\n",
      "  -0.22457527]\n",
      " [-0.26063493 -0.26063493 -0.26063493 ... -0.21891987 -0.21891987\n",
      "  -0.21891987]\n",
      " [-0.24077351 -0.24077351 -0.24077351 ... -0.24446118 -0.24446118\n",
      "  -0.24446118]\n",
      " ...\n",
      " [-0.19945773 -0.19945773 -0.19945773 ... -0.21475297 -0.21475297\n",
      "  -0.21475297]\n",
      " [-0.2226859  -0.2226859  -0.2226859  ... -0.21949983 -0.21949983\n",
      "  -0.21949983]\n",
      " [-0.25492153 -0.25492153 -0.25492153 ... -0.23160473 -0.23160473\n",
      "  -0.23160473]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24050513 -0.24050513 -0.24050513 ... -0.19125937 -0.19125937\n",
      "  -0.19125937]\n",
      " [-0.24115127 -0.24115127 -0.24115127 ... -0.27125758 -0.27125758\n",
      "  -0.27125758]\n",
      " [-0.26922867 -0.26922867 -0.26922867 ... -0.2017139  -0.2017139\n",
      "  -0.2017139 ]\n",
      " ...\n",
      " [-0.19779147 -0.19779147 -0.19779147 ... -0.26071554 -0.26071554\n",
      "  -0.26071554]\n",
      " [-0.26236874 -0.26236874 -0.26236874 ... -0.20335548 -0.20335548\n",
      "  -0.20335548]\n",
      " [-0.18621954 -0.18621954 -0.18621954 ... -0.22066852 -0.22066852\n",
      "  -0.22066852]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.22639608 -0.22639608 -0.22639608 ... -0.23796746 -0.23796746\n",
      "  -0.23796746]\n",
      " [-0.21431476 -0.21431476 -0.21431476 ... -0.20760016 -0.20760016\n",
      "  -0.20760016]\n",
      " [-0.17190574 -0.17190574 -0.17190574 ... -0.19151941 -0.19151941\n",
      "  -0.19151941]\n",
      " ...\n",
      " [-0.22708191 -0.22708191 -0.22708191 ... -0.22887377 -0.22887377\n",
      "  -0.22887377]\n",
      " [-0.2242764  -0.2242764  -0.2242764  ... -0.2168514  -0.2168514\n",
      "  -0.2168514 ]\n",
      " [-0.2077133  -0.2077133  -0.2077133  ... -0.23346624 -0.23346624\n",
      "  -0.23346624]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.14286406 -0.14286406 -0.14286406 ... -0.24154234 -0.24154234\n",
      "  -0.24154234]\n",
      " [-0.2383434  -0.2383434  -0.2383434  ... -0.22176783 -0.22176783\n",
      "  -0.22176783]\n",
      " [-0.22133443 -0.22133443 -0.22133443 ... -0.22663072 -0.22663072\n",
      "  -0.22663072]\n",
      " ...\n",
      " [-0.21863483 -0.21863483 -0.21863483 ... -0.2693412  -0.2693412\n",
      "  -0.2693412 ]\n",
      " [-0.19906658 -0.19906658 -0.19906658 ... -0.19492447 -0.19492447\n",
      "  -0.19492447]\n",
      " [-0.20900528 -0.20900528 -0.20900528 ... -0.21504733 -0.21504733\n",
      "  -0.21504733]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4545  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.27217075 -0.27217075 -0.27217075 ... -0.24030773 -0.24030773\n",
      "  -0.24030773]\n",
      " [-0.1994571  -0.1994571  -0.1994571  ... -0.23551035 -0.23551035\n",
      "  -0.23551035]\n",
      " [-0.18451492 -0.18451492 -0.18451492 ... -0.21639332 -0.21639332\n",
      "  -0.21639332]\n",
      " ...\n",
      " [-0.23753491 -0.23753491 -0.23753491 ... -0.1767325  -0.1767325\n",
      "  -0.1767325 ]\n",
      " [-0.22909665 -0.22909665 -0.22909665 ... -0.2288516  -0.2288516\n",
      "  -0.2288516 ]\n",
      " [-0.24536344 -0.24536344 -0.24536344 ... -0.20555954 -0.20555954\n",
      "  -0.20555954]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22293743 -0.22293743 -0.22293743 ... -0.232261   -0.232261\n",
      "  -0.232261  ]\n",
      " [-0.23475198 -0.23475198 -0.23475198 ... -0.22872305 -0.22872305\n",
      "  -0.22872305]\n",
      " [-0.1984874  -0.1984874  -0.1984874  ... -0.28776667 -0.28776667\n",
      "  -0.28776667]\n",
      " ...\n",
      " [-0.20428535 -0.20428535 -0.20428535 ... -0.21185479 -0.21185479\n",
      "  -0.21185479]\n",
      " [-0.14939255 -0.14939255 -0.14939255 ... -0.24622092 -0.24622092\n",
      "  -0.24622092]\n",
      " [-0.20546694 -0.20546694 -0.20546694 ... -0.22842205 -0.22842205\n",
      "  -0.22842205]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24643436 -0.24643436 -0.24643436 ... -0.22331022 -0.22331022\n",
      "  -0.22331022]\n",
      " [-0.2592883  -0.2592883  -0.2592883  ... -0.24272133 -0.24272133\n",
      "  -0.24272133]\n",
      " [-0.25255585 -0.25255585 -0.25255585 ... -0.2668097  -0.2668097\n",
      "  -0.2668097 ]\n",
      " ...\n",
      " [-0.27849555 -0.27849555 -0.27849555 ... -0.20454726 -0.20454726\n",
      "  -0.20454726]\n",
      " [-0.20908344 -0.20908344 -0.20908344 ... -0.20099103 -0.20099103\n",
      "  -0.20099103]\n",
      " [-0.22676751 -0.22676751 -0.22676751 ... -0.24685574 -0.24685574\n",
      "  -0.24685574]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22888741 -0.22888741 -0.22888741 ... -0.19327098 -0.19327098\n",
      "  -0.19327098]\n",
      " [-0.2856459  -0.2856459  -0.2856459  ... -0.21440667 -0.21440667\n",
      "  -0.21440667]\n",
      " [-0.2317571  -0.2317571  -0.2317571  ... -0.17918842 -0.17918842\n",
      "  -0.17918842]\n",
      " ...\n",
      " [-0.22950149 -0.22950149 -0.22950149 ... -0.25449073 -0.25449073\n",
      "  -0.25449073]\n",
      " [-0.23874128 -0.23874128 -0.23874128 ... -0.2608514  -0.2608514\n",
      "  -0.2608514 ]\n",
      " [-0.21130374 -0.21130374 -0.21130374 ... -0.23463482 -0.23463482\n",
      "  -0.23463482]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.25435647 -0.25435647 -0.25435647 ... -0.21289119 -0.21289119\n",
      "  -0.21289119]\n",
      " [-0.19415073 -0.19415073 -0.19415073 ... -0.25159422 -0.25159422\n",
      "  -0.25159422]\n",
      " [-0.21563005 -0.21563005 -0.21563005 ... -0.23469685 -0.23469685\n",
      "  -0.23469685]\n",
      " ...\n",
      " [-0.21481499 -0.21481499 -0.21481499 ... -0.2108469  -0.2108469\n",
      "  -0.2108469 ]\n",
      " [-0.22519135 -0.22519135 -0.22519135 ... -0.24998832 -0.24998832\n",
      "  -0.24998832]\n",
      " [-0.24305205 -0.24305205 -0.24305205 ... -0.25764638 -0.25764638\n",
      "  -0.25764638]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22229269 -0.22229269 -0.22229269 ... -0.23698376 -0.23698376\n",
      "  -0.23698376]\n",
      " [-0.22653651 -0.22653651 -0.22653651 ... -0.2476199  -0.2476199\n",
      "  -0.2476199 ]\n",
      " [-0.20908344 -0.20908344 -0.20908344 ... -0.28323558 -0.28323558\n",
      "  -0.28323558]\n",
      " ...\n",
      " [-0.21000099 -0.21000099 -0.21000099 ... -0.27785426 -0.27785426\n",
      "  -0.27785426]\n",
      " [-0.21623194 -0.21623194 -0.21623194 ... -0.24991825 -0.24991825\n",
      "  -0.24991825]\n",
      " [-0.2368798  -0.2368798  -0.2368798  ... -0.20955981 -0.20955981\n",
      "  -0.20955981]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23801062 -0.23801062 -0.23801062 ... -0.22636865 -0.22636865\n",
      "  -0.22636865]\n",
      " [-0.19475919 -0.19475919 -0.19475919 ... -0.19325982 -0.19325982\n",
      "  -0.19325982]\n",
      " [-0.20554978 -0.20554978 -0.20554978 ... -0.22977534 -0.22977534\n",
      "  -0.22977534]\n",
      " ...\n",
      " [-0.2387248  -0.2387248  -0.2387248  ... -0.22295126 -0.22295126\n",
      "  -0.22295126]\n",
      " [-0.18702477 -0.18702477 -0.18702477 ... -0.23198763 -0.23198763\n",
      "  -0.23198763]\n",
      " [-0.2524607  -0.2524607  -0.2524607  ... -0.23776403 -0.23776403\n",
      "  -0.23776403]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24171898 -0.24171898 -0.24171898 ... -0.23121753 -0.23121753\n",
      "  -0.23121753]\n",
      " [-0.27585158 -0.27585158 -0.27585158 ... -0.24054052 -0.24054052\n",
      "  -0.24054052]\n",
      " [-0.22240704 -0.22240704 -0.22240704 ... -0.19418164 -0.19418164\n",
      "  -0.19418164]\n",
      " ...\n",
      " [-0.2462701  -0.2462701  -0.2462701  ... -0.22947915 -0.22947915\n",
      "  -0.22947915]\n",
      " [-0.2302391  -0.2302391  -0.2302391  ... -0.24119721 -0.24119721\n",
      "  -0.24119721]\n",
      " [-0.26429296 -0.26429296 -0.26429296 ... -0.22848839 -0.22848839\n",
      "  -0.22848839]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4646  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2506857  -0.2506857  -0.2506857  ... -0.24258685 -0.24258685\n",
      "  -0.24258685]\n",
      " [-0.23874079 -0.23874079 -0.23874079 ... -0.23357567 -0.23357567\n",
      "  -0.23357567]\n",
      " [-0.25054348 -0.25054348 -0.25054348 ... -0.2560257  -0.2560257\n",
      "  -0.2560257 ]\n",
      " ...\n",
      " [-0.19397716 -0.19397716 -0.19397716 ... -0.22298238 -0.22298238\n",
      "  -0.22298238]\n",
      " [-0.21734405 -0.21734405 -0.21734405 ... -0.28984326 -0.28984326\n",
      "  -0.28984326]\n",
      " [-0.2283257  -0.2283257  -0.2283257  ... -0.20796359 -0.20796359\n",
      "  -0.20796359]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.273824   -0.273824   -0.273824   ... -0.23393229 -0.23393229\n",
      "  -0.23393229]\n",
      " [-0.30110642 -0.30110642 -0.30110642 ... -0.19389188 -0.19389188\n",
      "  -0.19389188]\n",
      " [-0.23306662 -0.23306662 -0.23306662 ... -0.22659378 -0.22659378\n",
      "  -0.22659378]\n",
      " ...\n",
      " [-0.2705903  -0.2705903  -0.2705903  ... -0.2759505  -0.2759505\n",
      "  -0.2759505 ]\n",
      " [-0.22831675 -0.22831675 -0.22831675 ... -0.25383002 -0.25383002\n",
      "  -0.25383002]\n",
      " [-0.18113673 -0.18113673 -0.18113673 ... -0.2584601  -0.2584601\n",
      "  -0.2584601 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.23403092 -0.23403092 -0.23403092 ... -0.24694364 -0.24694364\n",
      "  -0.24694364]\n",
      " [-0.26463804 -0.26463804 -0.26463804 ... -0.19614857 -0.19614857\n",
      "  -0.19614857]\n",
      " [-0.18392515 -0.18392515 -0.18392515 ... -0.23778138 -0.23778138\n",
      "  -0.23778138]\n",
      " ...\n",
      " [-0.25678244 -0.25678244 -0.25678244 ... -0.25494015 -0.25494015\n",
      "  -0.25494015]\n",
      " [-0.20063236 -0.20063236 -0.20063236 ... -0.22720699 -0.22720699\n",
      "  -0.22720699]\n",
      " [-0.23468736 -0.23468736 -0.23468736 ... -0.25438285 -0.25438285\n",
      "  -0.25438285]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.19559838 -0.19559838 -0.19559838 ... -0.20471586 -0.20471586\n",
      "  -0.20471586]\n",
      " [-0.23812069 -0.23812069 -0.23812069 ... -0.21862814 -0.21862814\n",
      "  -0.21862814]\n",
      " [-0.2556065  -0.2556065  -0.2556065  ... -0.23757014 -0.23757014\n",
      "  -0.23757014]\n",
      " ...\n",
      " [-0.23805396 -0.23805396 -0.23805396 ... -0.26428282 -0.26428282\n",
      "  -0.26428282]\n",
      " [-0.22804391 -0.22804391 -0.22804391 ... -0.22279796 -0.22279796\n",
      "  -0.22279796]\n",
      " [-0.25092104 -0.25092104 -0.25092104 ... -0.24290991 -0.24290991\n",
      "  -0.24290991]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.22543876 -0.22543876 -0.22543876 ... -0.23053543 -0.23053543\n",
      "  -0.23053543]\n",
      " [-0.2663606  -0.2663606  -0.2663606  ... -0.24494916 -0.24494916\n",
      "  -0.24494916]\n",
      " [-0.2087196  -0.2087196  -0.2087196  ... -0.25625402 -0.25625402\n",
      "  -0.25625402]\n",
      " ...\n",
      " [-0.2351336  -0.2351336  -0.2351336  ... -0.24896231 -0.24896231\n",
      "  -0.24896231]\n",
      " [-0.23828721 -0.23828721 -0.23828721 ... -0.30302948 -0.30302948\n",
      "  -0.30302948]\n",
      " [-0.26398286 -0.26398286 -0.26398286 ... -0.22848614 -0.22848614\n",
      "  -0.22848614]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24402103 -0.24402103 -0.24402103 ... -0.20474172 -0.20474172\n",
      "  -0.20474172]\n",
      " [-0.23009908 -0.23009908 -0.23009908 ... -0.21608531 -0.21608531\n",
      "  -0.21608531]\n",
      " [-0.24618314 -0.24618314 -0.24618314 ... -0.22316566 -0.22316566\n",
      "  -0.22316566]\n",
      " ...\n",
      " [-0.23506153 -0.23506153 -0.23506153 ... -0.20431197 -0.20431197\n",
      "  -0.20431197]\n",
      " [-0.27290025 -0.27290025 -0.27290025 ... -0.20350437 -0.20350437\n",
      "  -0.20350437]\n",
      " [-0.22526461 -0.22526461 -0.22526461 ... -0.2314245  -0.2314245\n",
      "  -0.2314245 ]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.18581004 -0.18581004 -0.18581004 ... -0.23893037 -0.23893037\n",
      "  -0.23893037]\n",
      " [-0.24517076 -0.24517076 -0.24517076 ... -0.24187626 -0.24187626\n",
      "  -0.24187626]\n",
      " [-0.19397716 -0.19397716 -0.19397716 ... -0.23603424 -0.23603424\n",
      "  -0.23603424]\n",
      " ...\n",
      " [-0.22525683 -0.22525683 -0.22525683 ... -0.19701271 -0.19701271\n",
      "  -0.19701271]\n",
      " [-0.211526   -0.211526   -0.211526   ... -0.24250337 -0.24250337\n",
      "  -0.24250337]\n",
      " [-0.24335226 -0.24335226 -0.24335226 ... -0.24435341 -0.24435341\n",
      "  -0.24435341]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2016926  -0.2016926  -0.2016926  ... -0.2555657  -0.2555657\n",
      "  -0.2555657 ]\n",
      " [-0.20284489 -0.20284489 -0.20284489 ... -0.17007872 -0.17007872\n",
      "  -0.17007872]\n",
      " [-0.26578462 -0.26578462 -0.26578462 ... -0.25979728 -0.25979728\n",
      "  -0.25979728]\n",
      " ...\n",
      " [-0.2425513  -0.2425513  -0.2425513  ... -0.21099865 -0.21099865\n",
      "  -0.21099865]\n",
      " [-0.26412868 -0.26412868 -0.26412868 ... -0.22056758 -0.22056758\n",
      "  -0.22056758]\n",
      " [-0.27551746 -0.27551746 -0.27551746 ... -0.21975122 -0.21975122\n",
      "  -0.21975122]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4747  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.25436354 -0.25436354 -0.25436354 ... -0.18862714 -0.18862714\n",
      "  -0.18862714]\n",
      " [-0.21292584 -0.21292584 -0.21292584 ... -0.2588241  -0.2588241\n",
      "  -0.2588241 ]\n",
      " [-0.26166508 -0.26166508 -0.26166508 ... -0.23471902 -0.23471902\n",
      "  -0.23471902]\n",
      " ...\n",
      " [-0.21457624 -0.21457624 -0.21457624 ... -0.2486689  -0.2486689\n",
      "  -0.2486689 ]\n",
      " [-0.25064167 -0.25064167 -0.25064167 ... -0.2182573  -0.2182573\n",
      "  -0.2182573 ]\n",
      " [-0.19326654 -0.19326654 -0.19326654 ... -0.25441462 -0.25441462\n",
      "  -0.25441462]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.21368472 -0.21368472 -0.21368472 ... -0.21957089 -0.21957089\n",
      "  -0.21957089]\n",
      " [-0.20821288 -0.20821288 -0.20821288 ... -0.25693816 -0.25693816\n",
      "  -0.25693816]\n",
      " [-0.2265822  -0.2265822  -0.2265822  ... -0.26107776 -0.26107776\n",
      "  -0.26107776]\n",
      " ...\n",
      " [-0.21669237 -0.21669237 -0.21669237 ... -0.18975168 -0.18975168\n",
      "  -0.18975168]\n",
      " [-0.31788492 -0.31788492 -0.31788492 ... -0.24224631 -0.24224631\n",
      "  -0.24224631]\n",
      " [-0.24482048 -0.24482048 -0.24482048 ... -0.25530013 -0.25530013\n",
      "  -0.25530013]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.20565562 -0.20565562 -0.20565562 ... -0.22563514 -0.22563514\n",
      "  -0.22563514]\n",
      " [-0.22093824 -0.22093824 -0.22093824 ... -0.23287714 -0.23287714\n",
      "  -0.23287714]\n",
      " [-0.2160837  -0.2160837  -0.2160837  ... -0.22202656 -0.22202656\n",
      "  -0.22202656]\n",
      " ...\n",
      " [-0.2940737  -0.2940737  -0.2940737  ... -0.2142081  -0.2142081\n",
      "  -0.2142081 ]\n",
      " [-0.22086768 -0.22086768 -0.22086768 ... -0.22769625 -0.22769625\n",
      "  -0.22769625]\n",
      " [-0.2808279  -0.2808279  -0.2808279  ... -0.2307517  -0.2307517\n",
      "  -0.2307517 ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.24678439 -0.24678439 -0.24678439 ... -0.23081668 -0.23081668\n",
      "  -0.23081668]\n",
      " [-0.24670699 -0.24670699 -0.24670699 ... -0.17371692 -0.17371692\n",
      "  -0.17371692]\n",
      " [-0.2498278  -0.2498278  -0.2498278  ... -0.22308394 -0.22308394\n",
      "  -0.22308394]\n",
      " ...\n",
      " [-0.2495927  -0.2495927  -0.2495927  ... -0.2411815  -0.2411815\n",
      "  -0.2411815 ]\n",
      " [-0.24257913 -0.24257913 -0.24257913 ... -0.30557275 -0.30557275\n",
      "  -0.30557275]\n",
      " [-0.23975417 -0.23975417 -0.23975417 ... -0.20874685 -0.20874685\n",
      "  -0.20874685]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.22355041 -0.22355041 -0.22355041 ... -0.23590778 -0.23590778\n",
      "  -0.23590778]\n",
      " [-0.27213672 -0.27213672 -0.27213672 ... -0.25591484 -0.25591484\n",
      "  -0.25591484]\n",
      " [-0.2795647  -0.2795647  -0.2795647  ... -0.24207711 -0.24207711\n",
      "  -0.24207711]\n",
      " ...\n",
      " [-0.2603398  -0.2603398  -0.2603398  ... -0.22873363 -0.22873363\n",
      "  -0.22873363]\n",
      " [-0.22631405 -0.22631405 -0.22631405 ... -0.25178182 -0.25178182\n",
      "  -0.25178182]\n",
      " [-0.26478034 -0.26478034 -0.26478034 ... -0.2674167  -0.2674167\n",
      "  -0.2674167 ]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.24682404 -0.24682404 -0.24682404 ... -0.27262288 -0.27262288\n",
      "  -0.27262288]\n",
      " [-0.24659042 -0.24659042 -0.24659042 ... -0.2297364  -0.2297364\n",
      "  -0.2297364 ]\n",
      " [-0.24147445 -0.24147445 -0.24147445 ... -0.25323045 -0.25323045\n",
      "  -0.25323045]\n",
      " ...\n",
      " [-0.20609336 -0.20609336 -0.20609336 ... -0.23688349 -0.23688349\n",
      "  -0.23688349]\n",
      " [-0.21292584 -0.21292584 -0.21292584 ... -0.27311987 -0.27311987\n",
      "  -0.27311987]\n",
      " [-0.2650329  -0.2650329  -0.2650329  ... -0.25205535 -0.25205535\n",
      "  -0.25205535]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2667989  -0.2667989  -0.2667989  ... -0.19051373 -0.19051373\n",
      "  -0.19051373]\n",
      " [-0.29555005 -0.29555005 -0.29555005 ... -0.22403826 -0.22403826\n",
      "  -0.22403826]\n",
      " [-0.22679822 -0.22679822 -0.22679822 ... -0.22748631 -0.22748631\n",
      "  -0.22748631]\n",
      " ...\n",
      " [-0.25642404 -0.25642404 -0.25642404 ... -0.22877173 -0.22877173\n",
      "  -0.22877173]\n",
      " [-0.23915124 -0.23915124 -0.23915124 ... -0.24888848 -0.24888848\n",
      "  -0.24888848]\n",
      " [-0.28399605 -0.28399605 -0.28399605 ... -0.27757925 -0.27757925\n",
      "  -0.27757925]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.22910392 -0.22910392 -0.22910392 ... -0.27682173 -0.27682173\n",
      "  -0.27682173]\n",
      " [-0.19258848 -0.19258848 -0.19258848 ... -0.24174458 -0.24174458\n",
      "  -0.24174458]\n",
      " [-0.2142097  -0.2142097  -0.2142097  ... -0.2709119  -0.2709119\n",
      "  -0.2709119 ]\n",
      " ...\n",
      " [-0.21862921 -0.21862921 -0.21862921 ... -0.26709154 -0.26709154\n",
      "  -0.26709154]\n",
      " [-0.23558031 -0.23558031 -0.23558031 ... -0.23573698 -0.23573698\n",
      "  -0.23573698]\n",
      " [-0.24144861 -0.24144861 -0.24144861 ... -0.21010122 -0.21010122\n",
      "  -0.21010122]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4848  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.2713079  -0.2713079  -0.2713079  ... -0.26356697 -0.26356697\n",
      "  -0.26356697]\n",
      " [-0.25490177 -0.25490177 -0.25490177 ... -0.23597845 -0.23597845\n",
      "  -0.23597845]\n",
      " [-0.24755025 -0.24755025 -0.24755025 ... -0.25115725 -0.25115725\n",
      "  -0.25115725]\n",
      " ...\n",
      " [-0.27916485 -0.27916485 -0.27916485 ... -0.2243191  -0.2243191\n",
      "  -0.2243191 ]\n",
      " [-0.2245642  -0.2245642  -0.2245642  ... -0.22358337 -0.22358337\n",
      "  -0.22358337]\n",
      " [-0.22672662 -0.22672662 -0.22672662 ... -0.20855631 -0.20855631\n",
      "  -0.20855631]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.23146635 -0.23146635 -0.23146635 ... -0.18826093 -0.18826093\n",
      "  -0.18826093]\n",
      " [-0.23167163 -0.23167163 -0.23167163 ... -0.20526779 -0.20526779\n",
      "  -0.20526779]\n",
      " [-0.20862785 -0.20862785 -0.20862785 ... -0.31833556 -0.31833556\n",
      "  -0.31833556]\n",
      " ...\n",
      " [-0.25639147 -0.25639147 -0.25639147 ... -0.21217555 -0.21217555\n",
      "  -0.21217555]\n",
      " [-0.28027526 -0.28027526 -0.28027526 ... -0.2630396  -0.2630396\n",
      "  -0.2630396 ]\n",
      " [-0.2621839  -0.2621839  -0.2621839  ... -0.23645352 -0.23645352\n",
      "  -0.23645352]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.21369702 -0.21369702 -0.21369702 ... -0.23393106 -0.23393106\n",
      "  -0.23393106]\n",
      " [-0.24403092 -0.24403092 -0.24403092 ... -0.2334948  -0.2334948\n",
      "  -0.2334948 ]\n",
      " [-0.21881303 -0.21881303 -0.21881303 ... -0.27112955 -0.27112955\n",
      "  -0.27112955]\n",
      " ...\n",
      " [-0.28190783 -0.28190783 -0.28190783 ... -0.25172096 -0.25172096\n",
      "  -0.25172096]\n",
      " [-0.20233458 -0.20233458 -0.20233458 ... -0.26612195 -0.26612195\n",
      "  -0.26612195]\n",
      " [-0.28002518 -0.28002518 -0.28002518 ... -0.22270764 -0.22270764\n",
      "  -0.22270764]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2521269  -0.2521269  -0.2521269  ... -0.21699093 -0.21699093\n",
      "  -0.21699093]\n",
      " [-0.25030077 -0.25030077 -0.25030077 ... -0.2221744  -0.2221744\n",
      "  -0.2221744 ]\n",
      " [-0.21316539 -0.21316539 -0.21316539 ... -0.23225065 -0.23225065\n",
      "  -0.23225065]\n",
      " ...\n",
      " [-0.21658285 -0.21658285 -0.21658285 ... -0.23378786 -0.23378786\n",
      "  -0.23378786]\n",
      " [-0.23268929 -0.23268929 -0.23268929 ... -0.17254259 -0.17254259\n",
      "  -0.17254259]\n",
      " [-0.2528356  -0.2528356  -0.2528356  ... -0.24298333 -0.24298333\n",
      "  -0.24298333]], shape=(8192, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.26984927 -0.26984927 -0.26984927 ... -0.24529219 -0.24529219\n",
      "  -0.24529219]\n",
      " [-0.25112462 -0.25112462 -0.25112462 ... -0.25174344 -0.25174344\n",
      "  -0.25174344]\n",
      " [-0.22615665 -0.22615665 -0.22615665 ... -0.23483276 -0.23483276\n",
      "  -0.23483276]\n",
      " ...\n",
      " [-0.23128834 -0.23128834 -0.23128834 ... -0.26323932 -0.26323932\n",
      "  -0.26323932]\n",
      " [-0.25968194 -0.25968194 -0.25968194 ... -0.25421366 -0.25421366\n",
      "  -0.25421366]\n",
      " [-0.22657561 -0.22657561 -0.22657561 ... -0.30627003 -0.30627003\n",
      "  -0.30627003]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.17424306 -0.17424306 -0.17424306 ... -0.23624444 -0.23624444\n",
      "  -0.23624444]\n",
      " [-0.21892571 -0.21892571 -0.21892571 ... -0.26961914 -0.26961914\n",
      "  -0.26961914]\n",
      " [-0.24676792 -0.24676792 -0.24676792 ... -0.22595768 -0.22595768\n",
      "  -0.22595768]\n",
      " ...\n",
      " [-0.26171774 -0.26171774 -0.26171774 ... -0.20032802 -0.20032802\n",
      "  -0.20032802]\n",
      " [-0.22521436 -0.22521436 -0.22521436 ... -0.22760205 -0.22760205\n",
      "  -0.22760205]\n",
      " [-0.28012815 -0.28012815 -0.28012815 ... -0.25938267 -0.25938267\n",
      "  -0.25938267]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.24286726 -0.24286726 -0.24286726 ... -0.2556029  -0.2556029\n",
      "  -0.2556029 ]\n",
      " [-0.27039343 -0.27039343 -0.27039343 ... -0.24743271 -0.24743271\n",
      "  -0.24743271]\n",
      " [-0.25420144 -0.25420144 -0.25420144 ... -0.22579297 -0.22579297\n",
      "  -0.22579297]\n",
      " ...\n",
      " [-0.24304068 -0.24304068 -0.24304068 ... -0.23833746 -0.23833746\n",
      "  -0.23833746]\n",
      " [-0.2512182  -0.2512182  -0.2512182  ... -0.22980455 -0.22980455\n",
      "  -0.22980455]\n",
      " [-0.25850046 -0.25850046 -0.25850046 ... -0.23364362 -0.23364362\n",
      "  -0.23364362]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.25625744 -0.25625744 -0.25625744 ... -0.2376902  -0.2376902\n",
      "  -0.2376902 ]\n",
      " [-0.31954476 -0.31954476 -0.31954476 ... -0.25070894 -0.25070894\n",
      "  -0.25070894]\n",
      " [-0.27458596 -0.27458596 -0.27458596 ... -0.27538764 -0.27538764\n",
      "  -0.27538764]\n",
      " ...\n",
      " [-0.2444016  -0.2444016  -0.2444016  ... -0.24238124 -0.24238124\n",
      "  -0.24238124]\n",
      " [-0.27140644 -0.27140644 -0.27140644 ... -0.2732959  -0.2732959\n",
      "  -0.2732959 ]\n",
      " [-0.24535844 -0.24535844 -0.24535844 ... -0.22945054 -0.22945054\n",
      "  -0.22945054]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "------------------------------------------------------------  step step   4949  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector\n",
      "[INFO]: embedding_vector\n",
      "  PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.23986994 -0.23986994 -0.23986994 ... -0.24169031 -0.24169031\n",
      "  -0.24169031]\n",
      " [-0.24237446 -0.24237446 -0.24237446 ... -0.23093976 -0.23093976\n",
      "  -0.23093976]\n",
      " [-0.23493694 -0.23493694 -0.23493694 ... -0.21154201 -0.21154201\n",
      "  -0.21154201]\n",
      " ...\n",
      " [-0.23662555 -0.23662555 -0.23662555 ... -0.24324594 -0.24324594\n",
      "  -0.24324594]\n",
      " [-0.24118835 -0.24118835 -0.24118835 ... -0.26387945 -0.26387945\n",
      "  -0.26387945]\n",
      " [-0.19845355 -0.19845355 -0.19845355 ... -0.24414644 -0.24414644\n",
      "  -0.24414644]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.25037906 -0.25037906 -0.25037906 ... -0.23343503 -0.23343503\n",
      "  -0.23343503]\n",
      " [-0.2629516  -0.2629516  -0.2629516  ... -0.27090853 -0.27090853\n",
      "  -0.27090853]\n",
      " [-0.261898   -0.261898   -0.261898   ... -0.23969518 -0.23969518\n",
      "  -0.23969518]\n",
      " ...\n",
      " [-0.18548758 -0.18548758 -0.18548758 ... -0.2840953  -0.2840953\n",
      "  -0.2840953 ]\n",
      " [-0.23644374 -0.23644374 -0.23644374 ... -0.23270768 -0.23270768\n",
      "  -0.23270768]\n",
      " [-0.27174586 -0.27174586 -0.27174586 ... -0.26489466 -0.26489466\n",
      "  -0.26489466]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.2999079  -0.2999079  -0.2999079  ... -0.23383988 -0.23383988\n",
      "  -0.23383988]\n",
      " [-0.23458661 -0.23458661 -0.23458661 ... -0.21408391 -0.21408391\n",
      "  -0.21408391]\n",
      " [-0.25493696 -0.25493696 -0.25493696 ... -0.26160455 -0.26160455\n",
      "  -0.26160455]\n",
      " ...\n",
      " [-0.26476446 -0.26476446 -0.26476446 ... -0.22313845 -0.22313845\n",
      "  -0.22313845]\n",
      " [-0.23379648 -0.23379648 -0.23379648 ... -0.24443206 -0.24443206\n",
      "  -0.24443206]\n",
      " [-0.24343008 -0.24343008 -0.24343008 ... -0.20665948 -0.20665948\n",
      "  -0.20665948]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.2531609  -0.2531609  -0.2531609  ... -0.24057162 -0.24057162\n",
      "  -0.24057162]\n",
      " [-0.25487775 -0.25487775 -0.25487775 ... -0.21782023 -0.21782023\n",
      "  -0.21782023]\n",
      " [-0.25217924 -0.25217924 -0.25217924 ... -0.22962268 -0.22962268\n",
      "  -0.22962268]\n",
      " ...\n",
      " [-0.3263282  -0.3263282  -0.3263282  ... -0.24834344 -0.24834344\n",
      "  -0.24834344]\n",
      " [-0.2694171  -0.2694171  -0.2694171  ... -0.27399072 -0.27399072\n",
      "  -0.27399072]\n",
      " [-0.21881284 -0.21881284 -0.21881284 ... -0.26389512 -0.26389512\n",
      "  -0.26389512]], shape=(8192, 40), dtype=float32)\n",
      "}PerReplica:{\n",
      "  0: tf.Tensor(\n",
      "[[-0.20687424 -0.20687424 -0.20687424 ... -0.2221736  -0.2221736\n",
      "  -0.2221736 ]\n",
      " [-0.25880358 -0.25880358 -0.25880358 ... -0.22520854 -0.22520854\n",
      "  -0.22520854]\n",
      " [-0.27138382 -0.27138382 -0.27138382 ... -0.23329027 -0.23329027\n",
      "  -0.23329027]\n",
      " ...\n",
      " [-0.2401509  -0.2401509  -0.2401509  ... -0.23446536 -0.23446536\n",
      "  -0.23446536]\n",
      " [-0.24403286 -0.24403286 -0.24403286 ... -0.27446926 -0.27446926\n",
      "  -0.27446926]\n",
      " [-0.23387225 -0.23387225 -0.23387225 ... -0.23907761 -0.23907761\n",
      "  -0.23907761]], shape=(8192, 40), dtype=float32),\n",
      "  1: tf.Tensor(\n",
      "[[-0.22344956 -0.22344956 -0.22344956 ... -0.22833571 -0.22833571\n",
      "  -0.22833571]\n",
      " [-0.23194185 -0.23194185 -0.23194185 ... -0.21191886 -0.21191886\n",
      "  -0.21191886]\n",
      " [-0.23020028 -0.23020028 -0.23020028 ... -0.22489744 -0.22489744\n",
      "  -0.22489744]\n",
      " ...\n",
      " [-0.25193235 -0.25193235 -0.25193235 ... -0.27135277 -0.27135277\n",
      "  -0.27135277]\n",
      " [-0.24686296 -0.24686296 -0.24686296 ... -0.26953402 -0.26953402\n",
      "  -0.26953402]\n",
      " [-0.26636225 -0.26636225 -0.26636225 ... -0.25624293 -0.25624293\n",
      "  -0.25624293]], shape=(8192, 40), dtype=float32),\n",
      "  2: tf.Tensor(\n",
      "[[-0.28136194 -0.28136194 -0.28136194 ... -0.24881288 -0.24881288\n",
      "  -0.24881288]\n",
      " [-0.26520234 -0.26520234 -0.26520234 ... -0.26157624 -0.26157624\n",
      "  -0.26157624]\n",
      " [-0.21311136 -0.21311136 -0.21311136 ... -0.2121953  -0.2121953\n",
      "  -0.2121953 ]\n",
      " ...\n",
      " [-0.24619848 -0.24619848 -0.24619848 ... -0.2550737  -0.2550737\n",
      "  -0.2550737 ]\n",
      " [-0.26096532 -0.26096532 -0.26096532 ... -0.24559599 -0.24559599\n",
      "  -0.24559599]\n",
      " [-0.20604698 -0.20604698 -0.20604698 ... -0.213556   -0.213556\n",
      "  -0.213556  ]], shape=(8192, 40), dtype=float32),\n",
      "  3: tf.Tensor(\n",
      "[[-0.25840718 -0.25840718 -0.25840718 ... -0.25522372 -0.25522372\n",
      "  -0.25522372]\n",
      " [-0.22320628 -0.22320628 -0.22320628 ... -0.2364969  -0.2364969\n",
      "  -0.2364969 ]\n",
      " [-0.2581673  -0.2581673  -0.2581673  ... -0.23624633 -0.23624633\n",
      "  -0.23624633]\n",
      " ...\n",
      " [-0.22897121 -0.22897121 -0.22897121 ... -0.2514059  -0.2514059\n",
      "  -0.2514059 ]\n",
      " [-0.25645512 -0.25645512 -0.25645512 ... -0.22097541 -0.22097541\n",
      "  -0.22097541]\n",
      " [-0.21199508 -0.21199508 -0.21199508 ... -0.25493002 -0.25493002\n",
      "  -0.25493002]], shape=(8192, 40), dtype=float32)\n",
      "}\n",
      "\n",
      "[INFO]: dumpped items to file ./sok_embedding_vectors_1.file\n",
      "[INFO]: dumpped items to file ./sok_embedding_vectors_0.file\n",
      "[INFO] loadded from file ./random_samples_0.file\n",
      "[INFO] loadded from file ./random_samples_0.file\n",
      "[INFO] loadded from file ./random_samples_1.file\n",
      "[INFO] loadded from file ./random_samples_1.file\n",
      "------------------------------ 0 ------------------------------\n",
      "------------------------------ 0 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 1 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.90052605 0.90052605 0.90052605 ... 0.9005313  0.9005313  0.9005313 ]\n",
      " [0.90055907 0.90055907 0.90055907 ... 0.9005337  0.9005337  0.9005337 ]\n",
      " [0.90046984 0.90046984 0.90046984 ... 0.9006437  0.9006437  0.9006437 ]\n",
      " ...\n",
      " [0.90049905 0.90049905 0.90049905 ... 0.90063065 0.90063065 0.90063065]\n",
      " [0.9005899  0.9005899  0.9005899  ... 0.9004804  0.9004804  0.9004804 ]\n",
      " [0.9005114  0.9005114  0.9005114  ... 0.9005455  0.9005455  0.9005455 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 2 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " ...\n",
      " [1. 1. 1. ... 1. 1. 1.]\n",
      " [1. 1. 1. ... 1. 1. 1.]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [1. 1. 1. ... 1. 1. 1.]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 1 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.8010717  0.8010717  0.8010717  ... 0.80135775 0.80135775 0.80135775]\n",
      " [0.8019426  0.8019426  0.8019426  ... 0.80218947 0.80218947 0.80218947]\n",
      " [0.8017676  0.8017676  0.8017676  ... 0.80201226 0.80201226 0.80201226]\n",
      " ...\n",
      " [0.80214393 0.80214393 0.80214393 ... 0.8014156  0.8014156  0.8014156 ]\n",
      " [0.8008634  0.8008634  0.8008634  ... 0.8021169  0.8021169  0.8021169 ]\n",
      " [0.80100465 0.80100465 0.80100465 ... 0.80109835 0.80109835 0.80109835]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.90052605 0.90052605 0.90052605 ... 0.9005313  0.9005313  0.9005313 ]\n",
      " [0.90055907 0.90055907 0.90055907 ... 0.9005337  0.9005337  0.9005337 ]\n",
      " [0.90046984 0.90046984 0.90046984 ... 0.9006437  0.9006437  0.9006437 ]\n",
      " ...\n",
      " [0.90049905 0.90049905 0.90049905 ... 0.90063065 0.90063065 0.90063065]\n",
      " [0.9005899  0.9005899  0.9005899  ... 0.9004804  0.9004804  0.9004804 ]\n",
      " [0.9005114  0.9005114  0.9005114  ... 0.9005455  0.9005455  0.9005455 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 3 ------------------------------\n",
      "------------------------------ 2 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.7029762  0.7029762  0.7029762  ... 0.70148593 0.70148593 0.70148593]\n",
      " [0.703499   0.703499   0.703499   ... 0.7024387  0.7024387  0.7024387 ]\n",
      " [0.70334077 0.70334077 0.70334077 ... 0.7089069  0.7089069  0.7089069 ]\n",
      " ...\n",
      " [0.7062126  0.7062126  0.7062126  ... 0.702146   0.702146   0.702146  ]\n",
      " [0.7034292  0.7034292  0.7034292  ... 0.7082195  0.7082195  0.7082195 ]\n",
      " [0.70296377 0.70296377 0.70296377 ... 0.70155084 0.70155084 0.70155084]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.8010717  0.8010717  0.8010717  ... 0.80135775 0.80135775 0.80135775]\n",
      " [0.8019426  0.8019426  0.8019426  ... 0.80218947 0.80218947 0.80218947]\n",
      " [0.8017676  0.8017676  0.8017676  ... 0.80201226 0.80201226 0.80201226]\n",
      " ...\n",
      " [0.80214393 0.80214393 0.80214393 ... 0.8014156  0.8014156  0.8014156 ]\n",
      " [0.8008634  0.8008634  0.8008634  ... 0.8021169  0.8021169  0.8021169 ]\n",
      " [0.80100465 0.80100465 0.80100465 ... 0.80109835 0.80109835 0.80109835]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 4 ------------------------------\n",
      "------------------------------ 3 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.6049563  0.6049563  0.6049563  ... 0.60658664 0.60658664 0.60658664]\n",
      " [0.60581076 0.60581076 0.60581076 ... 0.6055955  0.6055955  0.6055955 ]\n",
      " [0.60674024 0.60674024 0.60674024 ... 0.6066454  0.6066454  0.6066454 ]\n",
      " ...\n",
      " [0.60613    0.60613    0.60613    ... 0.6064851  0.6064851  0.6064851 ]\n",
      " [0.60596    0.60596    0.60596    ... 0.60520524 0.60520524 0.60520524]\n",
      " [0.60519755 0.60519755 0.60519755 ... 0.6082853  0.6082853  0.6082853 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.7029762  0.7029762  0.7029762  ... 0.70148593 0.70148593 0.70148593]\n",
      " [0.703499   0.703499   0.703499   ... 0.7024387  0.7024387  0.7024387 ]\n",
      " [0.70334077 0.70334077 0.70334077 ... 0.7089069  0.7089069  0.7089069 ]\n",
      " ...\n",
      " [0.7062126  0.7062126  0.7062126  ... 0.702146   0.702146   0.702146  ]\n",
      " [0.7034292  0.7034292  0.7034292  ... 0.7082195  0.7082195  0.7082195 ]\n",
      " [0.70296377 0.70296377 0.70296377 ... 0.70155084 0.70155084 0.70155084]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 5 ------------------------------\n",
      "------------------------------ 4 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.5128333  0.5128333  0.5128333  ... 0.5114437  0.5114437  0.5114437 ]\n",
      " [0.50756174 0.50756174 0.50756174 ... 0.51134473 0.51134473 0.51134473]\n",
      " [0.50510514 0.50510514 0.50510514 ... 0.5122752  0.5122752  0.5122752 ]\n",
      " ...\n",
      " [0.5048714  0.5048714  0.5048714  ... 0.5125397  0.5125397  0.5125397 ]\n",
      " [0.5107994  0.5107994  0.5107994  ... 0.506885   0.506885   0.506885  ]\n",
      " [0.5158305  0.5158305  0.5158305  ... 0.51701856 0.51701856 0.51701856]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.6049563  0.6049563  0.6049563  ... 0.60658664 0.60658664 0.60658664]\n",
      " [0.60581076 0.60581076 0.60581076 ... 0.6055955  0.6055955  0.6055955 ]\n",
      " [0.60674024 0.60674024 0.60674024 ... 0.6066454  0.6066454  0.6066454 ]\n",
      " ...\n",
      " [0.60613    0.60613    0.60613    ... 0.6064851  0.6064851  0.6064851 ]\n",
      " [0.60596    0.60596    0.60596    ... 0.60520524 0.60520524 0.60520524]\n",
      " [0.60519755 0.60519755 0.60519755 ... 0.6082853  0.6082853  0.6082853 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 5 ------------------------------\n",
      "------------------------------ 6 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.5128333  0.5128333  0.5128333  ... 0.5114437  0.5114437  0.5114437 ]\n",
      " [0.50756174 0.50756174 0.50756174 ... 0.51134473 0.51134473 0.51134473]\n",
      " [0.50510514 0.50510514 0.50510514 ... 0.5122752  0.5122752  0.5122752 ]\n",
      " ...\n",
      " [0.5048714  0.5048714  0.5048714  ... 0.5125397  0.5125397  0.5125397 ]\n",
      " [0.5107994  0.5107994  0.5107994  ... 0.506885   0.506885   0.506885  ]\n",
      " [0.5158305  0.5158305  0.5158305  ... 0.51701856 0.51701856 0.51701856]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.4185505  0.4185505  0.4185505  ... 0.41852283 0.41852283 0.41852283]\n",
      " [0.41285378 0.41285378 0.41285378 ... 0.41414076 0.41414076 0.41414076]\n",
      " [0.4216253  0.4216253  0.4216253  ... 0.41742593 0.41742593 0.41742593]\n",
      " ...\n",
      " [0.41351056 0.41351056 0.41351056 ... 0.4176001  0.4176001  0.4176001 ]\n",
      " [0.41477993 0.41477993 0.41477993 ... 0.41468954 0.41468954 0.41468954]\n",
      " [0.41689855 0.41689855 0.41689855 ... 0.41736287 0.41736287 0.41736287]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 6 ------------------------------\n",
      "------------------------------ 7 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.4185505  0.4185505  0.4185505  ... 0.41852283 0.41852283 0.41852283]\n",
      " [0.41285378 0.41285378 0.41285378 ... 0.41414076 0.41414076 0.41414076]\n",
      " [0.4216253  0.4216253  0.4216253  ... 0.41742593 0.41742593 0.41742593]\n",
      " ...\n",
      " [0.41351056 0.41351056 0.41351056 ... 0.4176001  0.4176001  0.4176001 ]\n",
      " [0.41477993 0.41477993 0.41477993 ... 0.41468954 0.41468954 0.41468954]\n",
      " [0.41689855 0.41689855 0.41689855 ... 0.41736287 0.41736287 0.41736287]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.32188094 0.32188094 0.32188094 ... 0.3332523  0.3332523  0.3332523 ]\n",
      " [0.32969487 0.32969487 0.32969487 ... 0.3214466  0.3214466  0.3214466 ]\n",
      " [0.32326683 0.32326683 0.32326683 ... 0.3335042  0.3335042  0.3335042 ]\n",
      " ...\n",
      " [0.32295296 0.32295296 0.32295296 ... 0.3325823  0.3325823  0.3325823 ]\n",
      " [0.3280143  0.3280143  0.3280143  ... 0.32656175 0.32656175 0.32656175]\n",
      " [0.33215436 0.33215436 0.33215436 ... 0.3178175  0.3178175  0.3178175 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 7 ------------------------------\n",
      "------------------------------ 8 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.32188094 0.32188094 0.32188094 ... 0.3332523  0.3332523  0.3332523 ]\n",
      " [0.32969487 0.32969487 0.32969487 ... 0.3214466  0.3214466  0.3214466 ]\n",
      " [0.32326683 0.32326683 0.32326683 ... 0.3335042  0.3335042  0.3335042 ]\n",
      " ...\n",
      " [0.32295296 0.32295296 0.32295296 ... 0.3325823  0.3325823  0.3325823 ]\n",
      " [0.3280143  0.3280143  0.3280143  ... 0.32656175 0.32656175 0.32656175]\n",
      " [0.33215436 0.33215436 0.33215436 ... 0.3178175  0.3178175  0.3178175 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " ------------------------------ 8tf.Tensor(\n",
      "[[0.23505837 0.23505837 0.23505837 ... 0.23979488 0.23979488 0.23979488]\n",
      " [0.24067074 0.24067074 0.24067074 ... 0.23883832 0.23883832 0.23883832]\n",
      " [0.23718816 0.23718816 0.23718816 ... 0.2408967  0.2408967  0.2408967 ]\n",
      " ...\n",
      " [0.24934453 0.24934453 0.24934453 ... 0.2331647  0.2331647  0.2331647 ]\n",
      " [0.2427354  0.2427354  0.2427354  ... 0.23341992 0.23341992 0.23341992]\n",
      " [0.25642774 0.25642774 0.25642774 ... 0.23338681 0.23338681 0.23338681]], shape=(65536, 40), dtype=float32) \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------\n",
      "------------------------------ 9 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.23505837 0.23505837 0.23505837 ... 0.23979488 0.23979488 0.23979488]\n",
      " [0.24067074 0.24067074 0.24067074 ... 0.23883832 0.23883832 0.23883832]\n",
      " [0.23718816 0.23718816 0.23718816 ... 0.2408967  0.2408967  0.2408967 ]\n",
      " ...\n",
      " [0.24934453 0.24934453 0.24934453 ... 0.2331647  0.2331647  0.2331647 ]\n",
      " [0.2427354  0.2427354  0.2427354  ... 0.23341992 0.23341992 0.23341992]\n",
      " [0.25642774 0.25642774 0.25642774 ... 0.23338681 0.23338681 0.23338681]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 9 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.16279979 0.16279979 0.16279979 ... 0.17118676 0.17118676 0.17118676]\n",
      " [0.16621807 0.16621807 0.16621807 ... 0.16611832 0.16611832 0.16611832]\n",
      " [0.1661632  0.1661632  0.1661632  ... 0.16512689 0.16512689 0.16512689]\n",
      " ...\n",
      " [0.17154846 0.17154846 0.17154846 ... 0.16619283 0.16619283 0.16619283]\n",
      " [0.15812562 0.15812562 0.15812562 ... 0.14438304 0.14438304 0.14438304]\n",
      " [0.15989152 0.15989152 0.15989152 ... 0.15028611 0.15028611 0.15028611]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.16279979 0.16279979 0.16279979 ... 0.17118676 0.17118676 0.17118676]\n",
      " [0.16621807 0.16621807 0.16621807 ... 0.16611832 0.16611832 0.16611832]\n",
      " [0.1661632  0.1661632  0.1661632  ... 0.16512689 0.16512689 0.16512689]\n",
      " ...\n",
      " [0.17154846 0.17154846 0.17154846 ... 0.16619283 0.16619283 0.16619283]\n",
      " [0.15812562 0.15812562 0.15812562 ... 0.14438304 0.14438304 0.14438304]\n",
      " [0.15989152 0.15989152 0.15989152 ... 0.15028611 0.15028611 0.15028611]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 10 ------------------------------\n",
      "------------------------------ 10 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.0968654  0.0968654  0.0968654  ... 0.06834027 0.06834027 0.06834027]\n",
      " [0.08394555 0.08394555 0.08394555 ... 0.08104148 0.08104148 0.08104148]\n",
      " [0.10349192 0.10349192 0.10349192 ... 0.08655374 0.08655374 0.08655374]\n",
      " ...\n",
      " [0.09681032 0.09681032 0.09681032 ... 0.09119104 0.09119104 0.09119104]\n",
      " [0.09105897 0.09105897 0.09105897 ... 0.07888062 0.07888062 0.07888062]\n",
      " [0.10630296 0.10630296 0.10630296 ... 0.08723153 0.08723153 0.08723153]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.0968654  0.0968654  0.0968654  ... 0.06834027 0.06834027 0.06834027]\n",
      " [0.08394555 0.08394555 0.08394555 ... 0.08104148 0.08104148 0.08104148]\n",
      " [0.10349192 0.10349192 0.10349192 ... 0.08655374 0.08655374 0.08655374]\n",
      " ...\n",
      " [0.09681032 0.09681032 0.09681032 ... 0.09119104 0.09119104 0.09119104]\n",
      " [0.09105897 0.09105897 0.09105897 ... 0.07888062 0.07888062 0.07888062]\n",
      " [0.10630296 0.10630296 0.10630296 ... 0.08723153 0.08723153 0.08723153]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 11 ------------------------------\n",
      "------------------------------ 11 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.03028748 0.03028748 0.03028748 ... 0.0204878  0.0204878  0.0204878 ]\n",
      " [0.04435753 0.04435753 0.04435753 ... 0.0152806  0.0152806  0.0152806 ]\n",
      " [0.01885913 0.01885913 0.01885913 ... 0.00630209 0.00630209 0.00630209]\n",
      " ...\n",
      " [0.03745253 0.03745253 0.03745253 ... 0.02543462 0.02543462 0.02543462]\n",
      " [0.03354985 0.03354985 0.03354985 ... 0.02504774 0.02504774 0.02504774]\n",
      " [0.03376191 0.03376191 0.03376191 ... 0.02583945 0.02583945 0.02583945]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[0.03028748 0.03028748 0.03028748 ... 0.0204878  0.0204878  0.0204878 ]\n",
      " [0.04435758 0.04435758 0.04435758 ... 0.0152806  0.0152806  0.0152806 ]\n",
      " [0.01885916 0.01885916 0.01885916 ... 0.00630209 0.00630209 0.00630209]\n",
      " ...\n",
      " [0.03745253 0.03745253 0.03745253 ... 0.02543462 0.02543462 0.02543462]\n",
      " [0.03354986 0.03354986 0.03354986 ... 0.02504774 0.02504774 0.02504774]\n",
      " [0.03376191 0.03376191 0.03376191 ... 0.02583945 0.02583945 0.02583945]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 12 ------------------------------\n",
      "------------------------------ 12 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.0312741  -0.0312741  -0.0312741  ... -0.03134615 -0.03134615\n",
      "  -0.03134615]\n",
      " [-0.02752096 -0.02752096 -0.02752096 ... -0.03708912 -0.03708912\n",
      "  -0.03708912]\n",
      " [-0.04079379 -0.04079379 -0.04079379 ... -0.048729   -0.048729\n",
      "  -0.048729  ]\n",
      " ...\n",
      " [-0.01866249 -0.01866249 -0.01866249 ... -0.02549717 -0.02549717\n",
      "  -0.02549717]\n",
      " [-0.02577814 -0.02577814 -0.02577814 ... -0.04188477 -0.04188477\n",
      "  -0.04188477]\n",
      " [-0.02924201 -0.02924201 -0.02924201 ... -0.03722169 -0.03722169\n",
      "  -0.03722169]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.0312741  -0.0312741  -0.0312741  ... -0.03134615 -0.03134615\n",
      "  -0.03134615]\n",
      " [-0.02752097 -0.02752097 -0.02752097 ... -0.03708914 -0.03708914\n",
      "  -0.03708914]\n",
      " [-0.04079379 -0.04079379 -0.04079379 ... -0.048729   -0.048729\n",
      "  -0.048729  ]\n",
      " ...\n",
      " [-0.01866249 -0.01866249 -0.01866249 ... -0.02549717 -0.02549717\n",
      "  -0.02549717]\n",
      " [-0.02577814 -0.02577814 -0.02577814 ... -0.04188477 -0.04188477\n",
      "  -0.04188477]\n",
      " [-0.02924201 -0.02924201 -0.02924201 ... -0.03722169 -0.03722169\n",
      "  -0.03722169]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 13 ------------------------------\n",
      "------------------------------ 13 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.06654122 -0.06654122 -0.06654122 ... -0.11854242 -0.11854242\n",
      "  -0.11854242]\n",
      " [-0.09573373 -0.09573373 -0.09573373 ... -0.08109956 -0.08109956\n",
      "  -0.08109956]\n",
      " [-0.07465851 -0.07465851 -0.07465851 ... -0.09946431 -0.09946431\n",
      "  -0.09946431]\n",
      " ...\n",
      " [-0.09270878 -0.09270878 -0.09270878 ... -0.08400823 -0.08400823\n",
      "  -0.08400823]\n",
      " [-0.08545051 -0.08545051 -0.08545051 ... -0.08018639 -0.08018639\n",
      "  -0.08018639]\n",
      " [-0.09431655 -0.09431655 -0.09431655 ... -0.09268773 -0.09268773\n",
      "  -0.09268773]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.06654122 -0.06654122 -0.06654122 ... -0.11854242 -0.11854242\n",
      "  -0.11854242]\n",
      " [-0.09573373 -0.09573373 -0.09573373 ... -0.08109956 -0.08109956\n",
      "  -0.08109956]\n",
      " [-0.07465851 -0.07465851 -0.07465851 ... -0.09946431 -0.09946431\n",
      "  -0.09946431]\n",
      " ...\n",
      " [-0.09270878 -0.09270878 -0.09270878 ... -0.08400823 -0.08400823\n",
      "  -0.08400823]\n",
      " [-0.08545051 -0.08545051 -0.08545051 ... -0.0801864  -0.0801864\n",
      "  -0.0801864 ]\n",
      " [-0.09431655 -0.09431655 -0.09431655 ... -0.09268776 -0.09268776\n",
      "  -0.09268776]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 14 ------------------------------\n",
      "------------------------------ 14 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.11253113 -0.11253113 -0.11253113 ... -0.12580246 -0.12580246\n",
      "  -0.12580246]\n",
      " [-0.11921576 -0.11921576 -0.11921576 ... -0.13080937 -0.13080937\n",
      "  -0.13080937]\n",
      " [-0.12867144 -0.12867144 -0.12867144 ... -0.11974896 -0.11974896\n",
      "  -0.11974896]\n",
      " ...\n",
      " [-0.10669142 -0.10669142 -0.10669142 ... -0.14308561 -0.14308561\n",
      "  -0.14308561]\n",
      " [-0.13619022 -0.13619022 -0.13619022 ... -0.11941485 -0.11941485\n",
      "  -0.11941485]\n",
      " [-0.12420082 -0.12420082 -0.12420082 ... -0.1297497  -0.1297497\n",
      "  -0.1297497 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " ------------------------------ tf.Tensor(\n",
      "[[-0.11253113 -0.11253113 -0.11253113 ... -0.12580246 -0.12580246\n",
      "  -0.12580246]\n",
      " [-0.11921576 -0.11921576 -0.11921576 ... -0.13080937 -0.13080937\n",
      "  -0.13080937]\n",
      " [-0.12867141 -0.12867141 -0.12867141 ... -0.11974896 -0.11974896\n",
      "  -0.11974896]\n",
      " ...\n",
      " [-0.10669142 -0.10669142 -0.10669142 ... -0.14308561 -0.14308561\n",
      "  -0.14308561]\n",
      " [-0.13619024 -0.13619024 -0.13619024 ... -0.1194149  -0.1194149\n",
      "  -0.1194149 ]\n",
      " [-0.1242008  -0.1242008  -0.1242008  ... -0.12974973 -0.12974973\n",
      "  -0.12974973]], shape=(65536, 40), dtype=float32)15\n",
      " ------------------------------\n",
      "------------------------------[INFO]: embedding_vector:\n",
      "  15 ------------------------------\n",
      "tf.Tensor(\n",
      "[[-0.17438808 -0.17438808 -0.17438808 ... -0.16145132 -0.16145132\n",
      "  -0.16145132]\n",
      " [-0.13499284 -0.13499284 -0.13499284 ... -0.15988389 -0.15988389\n",
      "  -0.15988389]\n",
      " [-0.16099232 -0.16099232 -0.16099232 ... -0.17622605 -0.17622605\n",
      "  -0.17622605]\n",
      " ...\n",
      " [-0.1669551  -0.1669551  -0.1669551  ... -0.16675307 -0.16675307\n",
      "  -0.16675307]\n",
      " [-0.1811203  -0.1811203  -0.1811203  ... -0.16348746 -0.16348746\n",
      "  -0.16348746]\n",
      " [-0.1748159  -0.1748159  -0.1748159  ... -0.1809609  -0.1809609\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  -0.1809609 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 16 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1743881  -0.1743881  -0.1743881  ... -0.16145132 -0.16145132\n",
      "  -0.16145132]\n",
      " [-0.13499284 -0.13499284 -0.13499284 ... -0.15988389 -0.15988389\n",
      "  -0.15988389]\n",
      " [-0.16099232 -0.16099232 -0.16099232 ... -0.17622605 -0.17622605\n",
      "  -0.17622605]\n",
      " ...\n",
      " [-0.1669551  -0.1669551  -0.1669551  ... -0.16675307 -0.16675307\n",
      "  -0.16675307]\n",
      " [-0.1811203  -0.1811203  -0.1811203  ... -0.16348746 -0.16348746\n",
      "  -0.16348746]\n",
      " [-0.1748159  -0.1748159  -0.1748159  ... -0.1809609  -0.1809609\n",
      "  -0.1809609 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.18558341 -0.18558341 -0.18558341 ... -0.22941414 -0.22941414\n",
      "  -0.22941414]\n",
      " [-0.19239876 -0.19239876 -0.19239876 ... -0.21053864 -0.21053864\n",
      "  -0.21053864]\n",
      " [-0.19028914 -0.19028914 -0.19028914 ... -0.2164091  -0.2164091\n",
      "  -0.2164091 ]\n",
      " ...\n",
      " [-0.17878489 -0.17878489 -0.17878489 ... -0.21031724 -0.21031724\n",
      "  -0.21031724]\n",
      " [-0.24041645 -0.24041645 -0.24041645 ... -0.21027762 -0.21027762\n",
      "  -0.21027762]\n",
      " [-0.22750843 -0.22750843 -0.22750843 ... -0.22710606 -0.22710606\n",
      "  -0.22710606]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 16 ------------------------------\n",
      "------------------------------ 17 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.18558341 -0.18558341 -0.18558341 ... -0.22941414 -0.22941414\n",
      "  -0.22941414]\n",
      " [-0.19239876 -0.19239876 -0.19239876 ... -0.21053864 -0.21053864\n",
      "  -0.21053864]\n",
      " [-0.19028914 -0.19028914 -0.19028914 ... -0.2164091  -0.2164091\n",
      "  -0.2164091 ]\n",
      " ...\n",
      " [-0.17878489 -0.17878489 -0.17878489 ... -0.21031724 -0.21031724\n",
      "  -0.21031724]\n",
      " [-0.24041645 -0.24041645 -0.24041645 ... -0.21027762 -0.21027762\n",
      "  -0.21027762]\n",
      " [-0.22750843 -0.22750843 -0.22750843 ... -0.22710608 -0.22710608\n",
      "  -0.22710608]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23731017 -0.23731017 -0.23731017 ... -0.25041643 -0.25041643\n",
      "  -0.25041643]\n",
      " [-0.2462632  -0.2462632  -0.2462632  ... -0.25780094 -0.25780094\n",
      "  -0.25780094]\n",
      " [-0.2618839  -0.2618839  -0.2618839  ... -0.2461817  -0.2461817\n",
      "  -0.2461817 ]\n",
      " ...\n",
      " [-0.24441311 -0.24441311 -0.24441311 ... -0.25110835 -0.25110835\n",
      "  -0.25110835]\n",
      " [-0.25490308 -0.25490308 -0.25490308 ... -0.25825828 -0.25825828\n",
      "  -0.25825828]\n",
      " [-0.25288773 -0.25288773 -0.25288773 ... -0.25172257 -0.25172257\n",
      "  -0.25172257]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------------------------------------  1718  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector:\n",
      " [INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23731016 -0.23731016 -0.23731016 ... -0.2504164  -0.2504164\n",
      "  -0.2504164 ]\n",
      " [-0.2462632  -0.2462632  -0.2462632  ... -0.25780094 -0.25780094\n",
      "  -0.25780094]\n",
      " [-0.2618839  -0.2618839  -0.2618839  ... -0.2461817  -0.2461817\n",
      "  -0.2461817 ]\n",
      " ...\n",
      " [-0.24441311 -0.24441311 -0.24441311 ... -0.25110832 -0.25110832\n",
      "  -0.25110832]\n",
      " [-0.25490308 -0.25490308 -0.25490308 ... -0.25825828 -0.25825828\n",
      "  -0.25825828]\n",
      " [-0.25288773 -0.25288773 -0.25288773 ... -0.25172257 -0.25172257\n",
      "  -0.25172257]], shape=(65536, 40), dtype=float32)tf.Tensor(\n",
      "[[-0.25777406 -0.25777406 -0.25777406 ... -0.25320214 -0.25320214\n",
      "  -0.25320214]\n",
      " [-0.2563972  -0.2563972  -0.2563972  ... -0.26466352 -0.26466352\n",
      "  -0.26466352]\n",
      " [-0.26688153 -0.26688153 -0.26688153 ... -0.2647105  -0.2647105\n",
      "  -0.2647105 ]\n",
      " ...\n",
      " [-0.25358817 -0.25358817 -0.25358817 ... -0.2617807  -0.2617807\n",
      "  -0.2617807 ]\n",
      " [-0.27929378 -0.27929378 -0.27929378 ... -0.2788083  -0.2788083\n",
      "  -0.2788083 ]\n",
      " [-0.3028677  -0.3028677  -0.3028677  ... -0.2739548  -0.2739548\n",
      "  -0.2739548 ]], shape=(65536, 40), dtype=float32)\n",
      "\n",
      "------------------------------ 19 ------------------------------\n",
      "------------------------------ 18 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.2896111  -0.2896111  -0.2896111  ... -0.28125638 -0.28125638\n",
      "  -0.28125638]\n",
      " [-0.30641434 -0.30641434 -0.30641434 ... -0.30045274 -0.30045274\n",
      "  -0.30045274]\n",
      " [-0.29099995 -0.29099995 -0.29099995 ... -0.2605071  -0.2605071\n",
      "  -0.2605071 ]\n",
      " ...\n",
      " [-0.30879536 -0.30879536 -0.30879536 ... -0.29969656 -0.29969656\n",
      "  -0.29969656]\n",
      " [-0.28311217 -0.28311217 -0.28311217 ... -0.28076622 -0.28076622\n",
      "  -0.28076622]\n",
      " [-0.3413444  -0.3413444  -0.3413444  ... -0.32740298 -0.32740298\n",
      "  -0.32740298]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.25777406 -0.25777406 -0.25777406 ... -0.25320214 -0.25320214\n",
      "  -0.25320214]\n",
      " [-0.2563972  -0.2563972  -0.2563972  ... -0.26466352 -0.26466352\n",
      "  -0.26466352]\n",
      " [-0.2668815  -0.2668815  -0.2668815  ... -0.2647105  -0.2647105\n",
      "  -0.2647105 ]\n",
      " ...\n",
      " [-0.25358817 -0.25358817 -0.25358817 ... -0.2617807  -0.2617807\n",
      "  -0.2617807 ]\n",
      " [-0.27929378 -0.27929378 -0.27929378 ... -0.2788083  -0.2788083\n",
      "  -0.2788083 ]\n",
      " [-0.3028677  -0.3028677  -0.3028677  ... -0.2739548  -0.2739548\n",
      "  -0.2739548 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 20 ------------------------------\n",
      "------------------------------ 19 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.30005008 -0.30005008 -0.30005008 ... -0.301296   -0.301296\n",
      "  -0.301296  ]\n",
      " [-0.27407846 -0.27407846 -0.27407846 ... -0.29747325 -0.29747325\n",
      "  -0.29747325]\n",
      " [-0.3280606  -0.3280606  -0.3280606  ... -0.32822445 -0.32822445\n",
      "  -0.32822445]\n",
      " ...\n",
      " [-0.31843626 -0.31843626 -0.31843626 ... -0.3075019  -0.3075019\n",
      "  -0.3075019 ]\n",
      " [-0.32108456 -0.32108456 -0.32108456 ... -0.28974748 -0.28974748\n",
      "  -0.28974748]\n",
      " [-0.3236766  -0.3236766  -0.3236766  ... -0.27159062 -0.27159062\n",
      "  -0.27159062]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.28961107 -0.28961107 -0.28961107 ... -0.28125635 -0.28125635\n",
      "  -0.28125635]\n",
      " [-0.30641428 -0.30641428 -0.30641428 ... -0.30045274 -0.30045274\n",
      "  -0.30045274]\n",
      " [-0.29099995 -0.29099995 -0.29099995 ... -0.26050708 -0.26050708\n",
      "  -0.26050708]\n",
      " ...\n",
      " [-0.3087954  -0.3087954  -0.3087954  ... -0.29969656 -0.29969656\n",
      "  -0.29969656]\n",
      " [-0.28311217 -0.28311217 -0.28311217 ... -0.28076622 -0.28076622\n",
      "  -0.28076622]\n",
      " [-0.34134436 -0.34134436 -0.34134436 ... -0.32740295 -0.32740295\n",
      "  -0.32740295]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 21 ------------------------------\n",
      "------------------------------ [INFO]: embedding_vector:\n",
      "20  ------------------------------\n",
      "tf.Tensor(\n",
      "[[-0.31901303 -0.31901303 -0.31901303 ... -0.31606743 -0.31606743\n",
      "  -0.31606743]\n",
      " [-0.3031985  -0.3031985  -0.3031985  ... -0.29684383 -0.29684383\n",
      "  -0.29684383]\n",
      " [-0.3093907  -0.3093907  -0.3093907  ... -0.32691437 -0.32691437\n",
      "  -0.32691437]\n",
      " ...\n",
      " [-0.31462306 -0.31462306 -0.31462306 ... -0.3337177  -0.3337177\n",
      "  -0.3337177 ]\n",
      " [-0.30985606 -0.30985606 -0.30985606 ... -0.30149662 -0.30149662\n",
      "  -0.30149662]\n",
      " [-0.30994755 -0.30994755 -0.30994755 ... -0.33786583 -0.33786583\n",
      "  -0.33786583]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 22 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.30005008 -0.30005008 -0.30005008 ... -0.301296   -0.301296\n",
      "  -0.301296  ]\n",
      " [-0.27407846 -0.27407846 -0.27407846 ... -0.29747325 -0.29747325\n",
      "  -0.29747325]\n",
      " [-0.3280606  -0.3280606  -0.3280606  ... -0.32822442 -0.32822442\n",
      "  -0.32822442]\n",
      " ...\n",
      " [-0.31843626 -0.31843626 -0.31843626 ... -0.30750188 -0.30750188\n",
      "  -0.30750188]\n",
      " [-0.32108456 -0.32108456 -0.32108456 ... -0.28974745 -0.28974745\n",
      "  -0.28974745]\n",
      " [-0.32367665 -0.32367665 -0.32367665 ... -0.2715906  -0.2715906\n",
      "  -0.2715906 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.32277828 -0.32277828 -0.32277828 ... -0.32983077 -0.32983077\n",
      "  -0.32983077]\n",
      " [-0.27947184 -0.27947184 -0.27947184 ... -0.2973408  -0.2973408\n",
      "  -0.2973408 ]\n",
      " [-0.31168154 -0.31168154 -0.31168154 ... -0.31368998 -0.31368998\n",
      "  -0.31368998]\n",
      " ...\n",
      " [-0.26390326 -0.26390326 -0.26390326 ... -0.304526   -0.304526\n",
      "  -0.304526  ]\n",
      " [-0.30593657 -0.30593657 -0.30593657 ... -0.30109614 -0.30109614\n",
      "  -0.30109614]\n",
      " [-0.28382862 -0.28382862 -0.28382862 ... -0.28956142 -0.28956142\n",
      "  -0.28956142]], shape=(65536, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 21 ------------------------------\n",
      "------------------------------ 23 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.319013   -0.319013   -0.319013   ... -0.31606743 -0.31606743\n",
      "  -0.31606743]\n",
      " [-0.3031985  -0.3031985  -0.3031985  ... -0.29684377 -0.29684377\n",
      "  -0.29684377]\n",
      " [-0.3093907  -0.3093907  -0.3093907  ... -0.3269143  -0.3269143\n",
      "  -0.3269143 ]\n",
      " ...\n",
      " [-0.31462306 -0.31462306 -0.31462306 ... -0.3337177  -0.3337177\n",
      "  -0.3337177 ]\n",
      " [-0.30985603 -0.30985603 -0.30985603 ... -0.3014966  -0.3014966\n",
      "  -0.3014966 ]\n",
      " [-0.30994752 -0.30994752 -0.30994752 ... -0.33786583 -0.33786583\n",
      "  -0.33786583]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.3010753  -0.3010753  -0.3010753  ... -0.26137453 -0.26137453\n",
      "  -0.26137453]\n",
      " [-0.2920072  -0.2920072  -0.2920072  ... -0.26624942 -0.26624942\n",
      "  -0.26624942]\n",
      " [-0.29443598 -0.29443598 -0.29443598 ... -0.31122905 -0.31122905\n",
      "  -0.31122905]\n",
      " ...\n",
      " [-0.2656673  -0.2656673  -0.2656673  ... -0.28852186 -0.28852186\n",
      "  -0.28852186]\n",
      " [-0.29285988 -0.29285988 -0.29285988 ... -0.28964502 -0.28964502\n",
      "  -0.28964502]\n",
      " [-0.2998341  -0.2998341  -0.2998341  ... -0.27875227 -0.27875227\n",
      "  -0.27875227]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 22 ------------------------------\n",
      "------------------------------ 24 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.32277828 -0.32277828 -0.32277828 ... -0.32983074 -0.32983074\n",
      "  -0.32983074]\n",
      " [-0.27947187 -0.27947187 -0.27947187 ... -0.2973408  -0.2973408\n",
      "  -0.2973408 ]\n",
      " [-0.31168154 -0.31168154 -0.31168154 ... -0.31368995 -0.31368995\n",
      "  -0.31368995]\n",
      " ...\n",
      " [-0.26390323 -0.26390323 -0.26390323 ... -0.304526   -0.304526\n",
      "  -0.304526  ]\n",
      " [-0.30593655 -0.30593655 -0.30593655 ... -0.30109614 -0.30109614\n",
      "  -0.30109614]\n",
      " [-0.28382865 -0.28382865 -0.28382865 ... -0.2895614  -0.2895614\n",
      "  -0.2895614 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.27834362 -0.27834362 -0.27834362 ... -0.26255178 -0.26255178\n",
      "  -0.26255178]\n",
      " [-0.285374   -0.285374   -0.285374   ... -0.29204327 -0.29204327\n",
      "  -0.29204327]\n",
      " [-0.30213454 -0.30213454 -0.30213454 ... -0.30334896 -0.30334896\n",
      "  -0.30334896]\n",
      " ...\n",
      " [-0.2540108  -0.2540108  -0.2540108  ... -0.289992   -0.289992\n",
      "  -0.289992  ]\n",
      " [-0.2605287  -0.2605287  -0.2605287  ... -0.28449565 -0.28449565\n",
      "  -0.28449565]\n",
      " [-0.26793268 -0.26793268 -0.26793268 ... -0.29072767 -0.29072767\n",
      "  -0.29072767]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 23 ------------------------------\n",
      "------------------------------ 25 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.3010753  -0.3010753  -0.3010753  ... -0.2613746  -0.2613746\n",
      "  -0.2613746 ]\n",
      " [-0.29200718 -0.29200718 -0.29200718 ... -0.26624942 -0.26624942\n",
      "  -0.26624942]\n",
      " [-0.29443598 -0.29443598 -0.29443598 ... -0.31122905 -0.31122905\n",
      "  -0.31122905]\n",
      " ...\n",
      " [-0.26566723 -0.26566723 -0.26566723 ... -0.28852186 -0.28852186\n",
      "  -0.28852186]\n",
      " [-0.29285985 -0.29285985 -0.29285985 ... -0.28964508 -0.28964508\n",
      "  -0.28964508]\n",
      " [-0.29983413 -0.29983413 -0.29983413 ... -0.27875227 -0.27875227\n",
      "  -0.27875227]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23426555 -0.23426555 -0.23426555 ... -0.22121611 -0.22121611\n",
      "  -0.22121611]\n",
      " [-0.2599339  -0.2599339  -0.2599339  ... -0.26001707 -0.26001707\n",
      "  -0.26001707]\n",
      " [-0.25415114 -0.25415114 -0.25415114 ... -0.24653983 -0.24653983\n",
      "  -0.24653983]\n",
      " ...\n",
      " [-0.2573175  -0.2573175  -0.2573175  ... -0.25539207 -0.25539207\n",
      "  -0.25539207]\n",
      " [-0.26618528 -0.26618528 -0.26618528 ... -0.25263914 -0.25263914\n",
      "  -0.25263914]\n",
      " [-0.24613588 -0.24613588 -0.24613588 ... -0.23047493 -0.23047493\n",
      "  -0.23047493]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------------------------------------  2426  ------------------------------------------------------------\n",
      "\n",
      "[INFO]: embedding_vector:\n",
      " [INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.2783436  -0.2783436  -0.2783436  ... -0.26255172 -0.26255172\n",
      "  -0.26255172]\n",
      " [-0.285374   -0.285374   -0.285374   ... -0.29204327 -0.29204327\n",
      "  -0.29204327]\n",
      " [-0.3021345  -0.3021345  -0.3021345  ... -0.3033489  -0.3033489\n",
      "  -0.3033489 ]\n",
      " ...\n",
      " [-0.2540108  -0.2540108  -0.2540108  ... -0.28999197 -0.28999197\n",
      "  -0.28999197]\n",
      " [-0.26052868 -0.26052868 -0.26052868 ... -0.28449565 -0.28449565\n",
      "  -0.28449565]\n",
      " [-0.26793268 -0.26793268 -0.26793268 ... -0.29072767 -0.29072767\n",
      "  -0.29072767]], shape=(65536, 40), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[-0.20332077 -0.20332077 -0.20332077 ... -0.24650502 -0.24650502\n",
      "  -0.24650502]\n",
      " [-0.26493365 -0.26493365 -0.26493365 ... -0.22278236 -0.22278236\n",
      "  -0.22278236]\n",
      " [-0.23291521 -0.23291521 -0.23291521 ... -0.2701033  -0.2701033\n",
      "  -0.2701033 ]\n",
      " ...\n",
      " [-0.279434   -0.279434   -0.279434   ... -0.25568685 -0.25568685\n",
      "  -0.25568685]\n",
      " [-0.22043912 -0.22043912 -0.22043912 ... -0.25438592 -0.25438592\n",
      "  -0.25438592]\n",
      " [-0.23791349 -0.23791349 -0.23791349 ... -0.22830802 -0.22830802\n",
      "  -0.22830802]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 27 ------------------------------\n",
      "------------------------------ 25 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20522626 -0.20522626 -0.20522626 ... -0.17386949 -0.17386949\n",
      "  -0.17386949]\n",
      " [-0.20582916 -0.20582916 -0.20582916 ... -0.1724671  -0.1724671\n",
      "  -0.1724671 ]\n",
      " [-0.21884425 -0.21884425 -0.21884425 ... -0.2164966  -0.2164966\n",
      "  -0.2164966 ]\n",
      " ...\n",
      " [-0.21075188 -0.21075188 -0.21075188 ... -0.21303985 -0.21303985\n",
      "  -0.21303985]\n",
      " [-0.23248424 -0.23248424 -0.23248424 ... -0.22775567 -0.22775567\n",
      "  -0.22775567]\n",
      " [-0.22323845 -0.22323845 -0.22323845 ... -0.22112778 -0.22112778\n",
      "  -0.22112778]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23426555 -0.23426555 -0.23426555 ... -0.22121608 -0.22121608\n",
      "  -0.22121608]\n",
      " [-0.2599339  -0.2599339  -0.2599339  ... -0.26001707 -0.26001707\n",
      "  -0.26001707]\n",
      " [-0.25415114 -0.25415114 -0.25415114 ... -0.24653985 -0.24653985\n",
      "  -0.24653985]\n",
      " ...\n",
      " [-0.2573175  -0.2573175  -0.2573175  ... -0.25539207 -0.25539207\n",
      "  -0.25539207]\n",
      " [-0.26618528 -0.26618528 -0.26618528 ... -0.25263914 -0.25263914\n",
      "  -0.25263914]\n",
      " [-0.24613589 -0.24613589 -0.24613589 ... -0.2304749  -0.2304749\n",
      "  -0.2304749 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 28 ------------------------------\n",
      "------------------------------ 26 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15625785 -0.15625785 -0.15625785 ... -0.2242238  -0.2242238\n",
      "  -0.2242238 ]\n",
      " [-0.1853661  -0.1853661  -0.1853661  ... -0.19528066 -0.19528066\n",
      "  -0.19528066]\n",
      " [-0.1635741  -0.1635741  -0.1635741  ... -0.2092087  -0.2092087\n",
      "  -0.2092087 ]\n",
      " ...\n",
      " [-0.22085923 -0.22085923 -0.22085923 ... -0.20258139 -0.20258139\n",
      "  -0.20258139]\n",
      " [-0.18809079 -0.18809079 -0.18809079 ... -0.22004685 -0.22004685\n",
      "  -0.22004685]\n",
      " [-0.20833743 -0.20833743 -0.20833743 ... -0.22157441 -0.22157441\n",
      "  -0.22157441]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20332073 -0.20332073 -0.20332073 ... -0.24650498 -0.24650498\n",
      "  -0.24650498]\n",
      " [-0.2649336  -0.2649336  -0.2649336  ... -0.22278233 -0.22278233\n",
      "  -0.22278233]\n",
      " [-0.23291521 -0.23291521 -0.23291521 ... -0.27010328 -0.27010328\n",
      "  -0.27010328]\n",
      " ...\n",
      " [-0.279434   -0.279434   -0.279434   ... -0.25568682 -0.25568682\n",
      "  -0.25568682]\n",
      " [-0.22043915 -0.22043915 -0.22043915 ... -0.25438592 -0.25438592\n",
      "  -0.25438592]\n",
      " [-0.23791349 -0.23791349 -0.23791349 ... -0.22830802 -0.22830802\n",
      "  -0.22830802]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 29 ------------------------------\n",
      "------------------------------ 27 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16556565 -0.16556565 -0.16556565 ... -0.1760715  -0.1760715\n",
      "  -0.1760715 ]\n",
      " [-0.17795168 -0.17795168 -0.17795168 ... -0.1632257  -0.1632257\n",
      "  -0.1632257 ]\n",
      " [-0.16011186 -0.16011186 -0.16011186 ... -0.18204345 -0.18204345\n",
      "  -0.18204345]\n",
      " ...\n",
      " [-0.19659984 -0.19659984 -0.19659984 ... -0.18282712 -0.18282712\n",
      "  -0.18282712]\n",
      " [-0.20446014 -0.20446014 -0.20446014 ... -0.18355143 -0.18355143\n",
      "  -0.18355143]\n",
      " [-0.2008438  -0.2008438  -0.2008438  ... -0.18044093 -0.18044093\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  -0.18044093]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      "------------------------------  30 tf.Tensor(\n",
      "[[-0.20522626 -0.20522626 -0.20522626 ... -0.17386943 -0.17386943\n",
      "  -0.17386943]\n",
      " [-0.2058292  -0.2058292  -0.2058292  ... -0.17246705 -0.17246705\n",
      "  -0.17246705]\n",
      " [-0.21884426 -0.21884426 -0.21884426 ... -0.21649659 -0.21649659\n",
      "  -0.21649659]\n",
      " ...\n",
      " [-0.21075189 -0.21075189 -0.21075189 ... -0.21303983 -0.21303983\n",
      "  -0.21303983]\n",
      " [-0.23248424 -0.23248424 -0.23248424 ... -0.22775568 -0.22775568\n",
      "  -0.22775568]\n",
      " [-0.22323845 -0.22323845 -0.22323845 ... -0.22112778 -0.22112778\n",
      "  -0.22112778]], shape=(65536, 40), dtype=float32)------------------------------\n",
      "\n",
      "------------------------------ 28 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.19175145 -0.19175145 -0.19175145 ... -0.16623998 -0.16623998\n",
      "  -0.16623998]\n",
      " [-0.16289337 -0.16289337 -0.16289337 ... -0.1725828  -0.1725828\n",
      "  -0.1725828 ]\n",
      " [-0.16131778 -0.16131778 -0.16131778 ... -0.16490965 -0.16490965\n",
      "  -0.16490965]\n",
      " ...\n",
      " [-0.18135208 -0.18135208 -0.18135208 ... -0.17282051 -0.17282051\n",
      "  -0.17282051]\n",
      " [-0.18483508 -0.18483508 -0.18483508 ... -0.12120736 -0.12120736\n",
      "  -0.12120736]\n",
      " [-0.18707147 -0.18707147 -0.18707147 ... -0.17588836 -0.17588836\n",
      "  -0.17588836]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 31 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15625785 -0.15625785 -0.15625785 ... -0.2242238  -0.2242238\n",
      "  -0.2242238 ]\n",
      " [-0.18536611 -0.18536611 -0.18536611 ... -0.19528066 -0.19528066\n",
      "  -0.19528066]\n",
      " [-0.16357401 -0.16357401 -0.16357401 ... -0.20920865 -0.20920865\n",
      "  -0.20920865]\n",
      " ...\n",
      " [-0.22085923 -0.22085923 -0.22085923 ... -0.20258139 -0.20258139\n",
      "  -0.20258139]\n",
      " [-0.1880908  -0.1880908  -0.1880908  ... -0.22004683 -0.22004683\n",
      "  -0.22004683]\n",
      " [-0.20833746 -0.20833746 -0.20833746 ... -0.22157437 -0.22157437\n",
      "  -0.22157437]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13461539 -0.13461539 -0.13461539 ... -0.1862741  -0.1862741\n",
      "  -0.1862741 ]\n",
      " [-0.16812241 -0.16812241 -0.16812241 ... -0.13700072 -0.13700072\n",
      "  -0.13700072]\n",
      " [-0.12897344 -0.12897344 -0.12897344 ... -0.18562101 -0.18562101\n",
      "  -0.18562101]\n",
      " ...\n",
      " [-0.15796353 -0.15796353 -0.15796353 ... -0.15093176 -0.15093176\n",
      "  -0.15093176]\n",
      " [-0.10758133 -0.10758133 -0.10758133 ... -0.15098229 -0.15098229\n",
      "  -0.15098229]\n",
      " [-0.13896157 -0.13896157 -0.13896157 ... -0.18083091 -0.18083091\n",
      "  -0.18083091]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 29 ------------------------------\n",
      "------------------------------ 32 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16556573 -0.16556573 -0.16556573 ... -0.1760715  -0.1760715\n",
      "  -0.1760715 ]\n",
      " [-0.17795168 -0.17795168 -0.17795168 ... -0.1632257  -0.1632257\n",
      "  -0.1632257 ]\n",
      " [-0.1601119  -0.1601119  -0.1601119  ... -0.18204343 -0.18204343\n",
      "  -0.18204343]\n",
      " ...\n",
      " [-0.19659984 -0.19659984 -0.19659984 ... -0.18282712 -0.18282712\n",
      "  -0.18282712]\n",
      " [-0.20446011 -0.20446011 -0.20446011 ... -0.18355134 -0.18355134\n",
      "  -0.18355134]\n",
      " [-0.20084386 -0.20084386 -0.20084386 ... -0.18044093 -0.18044093\n",
      "  -0.18044093]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16815904 -0.16815904 -0.16815904 ... -0.14582174 -0.14582174\n",
      "  -0.14582174]\n",
      " [-0.13498616 -0.13498616 -0.13498616 ... -0.13761719 -0.13761719\n",
      "  -0.13761719]\n",
      " [-0.15990289 -0.15990289 -0.15990289 ... -0.14938161 -0.14938161\n",
      "  -0.14938161]\n",
      " ...\n",
      " [-0.13369346 -0.13369346 -0.13369346 ... -0.17213689 -0.17213689\n",
      "  -0.17213689]\n",
      " [-0.17739835 -0.17739835 -0.17739835 ... -0.1543947  -0.1543947\n",
      "  -0.1543947 ]\n",
      " [-0.15795211 -0.15795211 -0.15795211 ... -0.12829769 -0.12829769\n",
      "  -0.12829769]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 30 ------------------------------\n",
      "------------------------------ 33 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.19175158 -0.19175158 -0.19175158 ... -0.16623998 -0.16623998\n",
      "  -0.16623998]\n",
      " [-0.16289338 -0.16289338 -0.16289338 ... -0.1725828  -0.1725828\n",
      "  -0.1725828 ]\n",
      " [-0.16131777 -0.16131777 -0.16131777 ... -0.16490963 -0.16490963\n",
      "  -0.16490963]\n",
      " ...\n",
      " [-0.18135199 -0.18135199 -0.18135199 ... -0.17282048 -0.17282048\n",
      "  -0.17282048]\n",
      " [-0.18483505 -0.18483505 -0.18483505 ... -0.12120731 -0.12120731\n",
      "  -0.12120731]\n",
      " [-0.18707147 -0.18707147 -0.18707147 ... -0.17588836 -0.17588836\n",
      "  -0.17588836]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15180266 -0.15180266 -0.15180266 ... -0.15956403 -0.15956403\n",
      "  -0.15956403]\n",
      " [-0.18056564 -0.18056564 -0.18056564 ... -0.11601669 -0.11601669\n",
      "  -0.11601669]\n",
      " [-0.10885718 -0.10885718 -0.10885718 ... -0.1777749  -0.1777749\n",
      "  -0.1777749 ]\n",
      " ...\n",
      " [-0.12706444 -0.12706444 -0.12706444 ... -0.12976488 -0.12976488\n",
      "  -0.12976488]\n",
      " [-0.1279442  -0.1279442  -0.1279442  ... -0.13762492 -0.13762492\n",
      "  -0.13762492]\n",
      " [-0.11606186 -0.11606186 -0.11606186 ... -0.15761319 -0.15761319\n",
      "  -0.15761319]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 31 ------------------------------\n",
      "------------------------------ 34 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.13461545 -0.13461545 -0.13461545 ... -0.1862741  -0.1862741\n",
      "  -0.1862741 ]\n",
      " [-0.16812243 -0.16812243 -0.16812243 ... -0.13700067 -0.13700067\n",
      "  -0.13700067]\n",
      " [-0.12897345 -0.12897345 -0.12897345 ... -0.185621   -0.185621\n",
      "  -0.185621  ]\n",
      " ...\n",
      " [-0.15796356 -0.15796356 -0.15796356 ... -0.15093173 -0.15093173\n",
      "  -0.15093173]\n",
      " [-0.10758135 -0.10758135 -0.10758135 ... -0.15098226 -0.15098226\n",
      "  -0.15098226]\n",
      " [-0.13896158 -0.13896158 -0.13896158 ... -0.18083091 -0.18083091\n",
      "  -0.18083091]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.14748009 -0.14748009 -0.14748009 ... -0.1309833  -0.1309833\n",
      "  -0.1309833 ]\n",
      " [-0.13158172 -0.13158172 -0.13158172 ... -0.130242   -0.130242\n",
      "  -0.130242  ]\n",
      " [-0.1297937  -0.1297937  -0.1297937  ... -0.11079911 -0.11079911\n",
      "  -0.11079911]\n",
      " ...\n",
      " [-0.15998903 -0.15998903 -0.15998903 ... -0.13199686 -0.13199686\n",
      "  -0.13199686]\n",
      " [-0.14270656 -0.14270656 -0.14270656 ... -0.15084621 -0.15084621\n",
      "  -0.15084621]\n",
      " [-0.15990183 -0.15990183 -0.15990183 ... -0.14487754 -0.14487754\n",
      "  -0.14487754]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 35 ------------------------------\n",
      "------------------------------ 32 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " [INFO]: embedding_vector:\n",
      "tf.Tensor(\n",
      "[[-0.10709302 -0.10709302 -0.10709302 ... -0.1116617  -0.1116617\n",
      "  -0.1116617 ]\n",
      " [-0.17265542 -0.17265542 -0.17265542 ... -0.16045204 -0.16045204\n",
      "  -0.16045204]\n",
      " [-0.12276509 -0.12276509 -0.12276509 ... -0.13697231 -0.13697231\n",
      "  -0.13697231]\n",
      " ...\n",
      " [-0.12169251 -0.12169251 -0.12169251 ... -0.13384937 -0.13384937\n",
      "  -0.13384937]\n",
      " [-0.10070504 -0.10070504 -0.10070504 ... -0.12427375 -0.12427375\n",
      "  -0.12427375]\n",
      " [-0.11460225 -0.11460225 -0.11460225 ... -0.1497563  -0.1497563\n",
      "  -0.1497563 ]], shape=(65536, 40), dtype=float32) \n",
      "tf.Tensor(\n",
      "[[-0.168159   -0.168159   -0.168159   ... -0.14582169 -0.14582169\n",
      "  -0.14582169]\n",
      " [-0.13498619 -0.13498619 -0.13498619 ... -0.13761713 -0.13761713\n",
      "  -0.13761713]\n",
      " [-0.15990293 -0.15990293 -0.15990293 ... -0.14938156 -0.14938156\n",
      "  -0.14938156]\n",
      " ...\n",
      " [-0.13369349 -0.13369349 -0.13369349 ... -0.17213684 -0.17213684\n",
      "  -0.17213684]\n",
      " [-0.17739835 -0.17739835 -0.17739835 ... -0.15439466 -0.15439466\n",
      "  -0.15439466]\n",
      " [-0.15795214 -0.15795214 -0.15795214 ... -0.12829767 -0.12829767\n",
      "  -0.12829767]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 36 ------------------------------\n",
      "------------------------------ 33 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1305122  -0.1305122  -0.1305122  ... -0.12057861 -0.12057861\n",
      "  -0.12057861]\n",
      " [-0.13115254 -0.13115254 -0.13115254 ... -0.17485873 -0.17485873\n",
      "  -0.17485873]\n",
      " [-0.12738791 -0.12738791 -0.12738791 ... -0.12302482 -0.12302482\n",
      "  -0.12302482]\n",
      " ...\n",
      " [-0.1447284  -0.1447284  -0.1447284  ... -0.14561641 -0.14561641\n",
      "  -0.14561641]\n",
      " [-0.14079258 -0.14079258 -0.14079258 ... -0.13417713 -0.13417713\n",
      "  -0.13417713]\n",
      " [-0.16059375 -0.16059375 -0.16059375 ... -0.11739247 -0.11739247\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  -0.11739247]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.15180266 -0.15180266 -0.15180266 ... -0.15956396 -0.15956396\n",
      "  -0.15956396]\n",
      " [-0.18056567 -0.18056567 -0.18056567 ... -0.11601666 -0.11601666\n",
      "  -0.11601666]\n",
      " [-0.10885726 -0.10885726 -0.10885726 ... -0.17777488 -0.17777488\n",
      "  -0.17777488]\n",
      " ...\n",
      " [-0.12706445 -0.12706445 -0.12706445 ... -0.12976485 -0.12976485\n",
      "  -0.12976485]\n",
      " [-0.12794422 -0.12794422 -0.12794422 ... -0.13762495 -0.13762495\n",
      "  -0.13762495]\n",
      " [-0.11606186 -0.11606186 -0.11606186 ... -0.1576132  -0.1576132\n",
      "  -0.1576132 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 37 ------------------------------\n",
      "------------------------------ 34 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.11994702 -0.11994702 -0.11994702 ... -0.16632766 -0.16632766\n",
      "  -0.16632766]\n",
      " [-0.12410761 -0.12410761 -0.12410761 ... -0.17561823 -0.17561823\n",
      "  -0.17561823]\n",
      " [-0.13765712 -0.13765712 -0.13765712 ... -0.15519011 -0.15519011\n",
      "  -0.15519011]\n",
      " ...\n",
      " [-0.14410038 -0.14410038 -0.14410038 ... -0.14400674 -0.14400674\n",
      "  -0.14400674]\n",
      " [-0.16299517 -0.16299517 -0.16299517 ... -0.12951873 -0.12951873\n",
      "  -0.12951873]\n",
      " [-0.17195132 -0.17195132 -0.17195132 ... -0.1433245  -0.1433245\n",
      "  -0.1433245 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.14748009 -0.14748009 -0.14748009 ... -0.1309832  -0.1309832\n",
      "  -0.1309832 ]\n",
      " [-0.13158178 -0.13158178 -0.13158178 ... -0.13024199 -0.13024199\n",
      "  -0.13024199]\n",
      " [-0.12979375 -0.12979375 -0.12979375 ... -0.11079908 -0.11079908\n",
      "  -0.11079908]\n",
      " ...\n",
      " [-0.15998904 -0.15998904 -0.15998904 ... -0.13199683 -0.13199683\n",
      "  -0.13199683]\n",
      " [-0.14270662 -0.14270662 -0.14270662 ... -0.15084623 -0.15084623\n",
      "  -0.15084623]\n",
      " [-0.15990186 -0.15990186 -0.15990186 ... -0.14487754 -0.14487754\n",
      "  -0.14487754]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 38 ------------------------------\n",
      "------------------------------ 35 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16781557 -0.16781557 -0.16781557 ... -0.17625841 -0.17625841\n",
      "  -0.17625841]\n",
      " [-0.13975957 -0.13975957 -0.13975957 ... -0.13837099 -0.13837099\n",
      "  -0.13837099]\n",
      " [-0.12085912 -0.12085912 -0.12085912 ... -0.15863958 -0.15863958\n",
      "  -0.15863958]\n",
      " ...\n",
      " [-0.17922544 -0.17922544 -0.17922544 ... -0.13088962 -0.13088962\n",
      "  -0.13088962]\n",
      " [-0.1706382  -0.1706382  -0.1706382  ... -0.13989025 -0.13989025\n",
      "  -0.13989025]\n",
      " [-0.17062347 -0.17062347 -0.17062347 ... -0.15872523 -0.15872523\n",
      "  -0.15872523]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 39 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.10709295 -0.10709295 -0.10709295 ... -0.11166165 -0.11166165\n",
      "  -0.11166165]\n",
      " [-0.1726554  -0.1726554  -0.1726554  ... -0.16045205 -0.16045205\n",
      "  -0.16045205]\n",
      " [-0.12276515 -0.12276515 -0.12276515 ... -0.13697225 -0.13697225\n",
      "  -0.13697225]\n",
      " ...\n",
      " [-0.12169252 -0.12169252 -0.12169252 ... -0.13384934 -0.13384934\n",
      "  -0.13384934]\n",
      " [-0.10070509 -0.10070509 -0.10070509 ... -0.12427374 -0.12427374\n",
      "  -0.12427374]\n",
      " [-0.11460223 -0.11460223 -0.11460223 ... -0.14975628 -0.14975628\n",
      "  -0.14975628]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 36 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1648547  -0.1648547  -0.1648547  ... -0.20128839 -0.20128839\n",
      "  -0.20128839]\n",
      " [-0.16366622 -0.16366622 -0.16366622 ... -0.13152812 -0.13152812\n",
      "  -0.13152812]\n",
      " [-0.15201205 -0.15201205 -0.15201205 ... -0.14416568 -0.14416568\n",
      "  -0.14416568]\n",
      " ...\n",
      " [-0.20643246 -0.20643246 -0.20643246 ... -0.16895425 -0.16895425\n",
      "  -0.16895425]\n",
      " [-0.17918105 -0.17918105 -0.17918105 ... -0.1814175  -0.1814175\n",
      "  -0.1814175 ]\n",
      " [-0.15831329 -0.15831329 -0.15831329 ... -0.15296361 -0.15296361\n",
      "  -0.15296361]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 40 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1305122  -0.1305122  -0.1305122  ... -0.12057861 -0.12057861\n",
      "  -0.12057861]\n",
      " [-0.13115251 -0.13115251 -0.13115251 ... -0.1748587  -0.1748587\n",
      "  -0.1748587 ]\n",
      " [-0.12738796 -0.12738796 -0.12738796 ... -0.12302484 -0.12302484\n",
      "  -0.12302484]\n",
      " ...\n",
      " [-0.1447284  -0.1447284  -0.1447284  ... -0.14561644 -0.14561644\n",
      "  -0.14561644]\n",
      " [-0.14079258 -0.14079258 -0.14079258 ... -0.13417709 -0.13417709\n",
      "  -0.13417709]\n",
      " [-0.16059372 -0.16059372 -0.16059372 ... -0.11739244 -0.11739244\n",
      "  -0.11739244]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 37 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.18479168 -0.18479168 -0.18479168 ... -0.18993594 -0.18993594\n",
      "  -0.18993594]\n",
      " [-0.17649812 -0.17649812 -0.17649812 ... -0.17080042 -0.17080042\n",
      "  -0.17080042]\n",
      " [-0.18804726 -0.18804726 -0.18804726 ... -0.15337944 -0.15337944\n",
      "  -0.15337944]\n",
      " ...\n",
      " [-0.16575228 -0.16575228 -0.16575228 ... -0.1668956  -0.1668956\n",
      "  -0.1668956 ]\n",
      " [-0.16589028 -0.16589028 -0.16589028 ... -0.19078445 -0.19078445\n",
      "  -0.19078445]\n",
      " [-0.0958271  -0.0958271  -0.0958271  ... -0.1520966  -0.1520966\n",
      "  -0.1520966 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      "------------------------------  41 ------------------------------tf.Tensor(\n",
      "[[-0.11994705 -0.11994705 -0.11994705 ... -0.16632757 -0.16632757\n",
      "  -0.16632757]\n",
      " [-0.12410764 -0.12410764 -0.12410764 ... -0.1756182  -0.1756182\n",
      "  -0.1756182 ]\n",
      " [-0.13765718 -0.13765718 -0.13765718 ... -0.15519008 -0.15519008\n",
      "  -0.15519008]\n",
      " ...\n",
      " [-0.14410043 -0.14410043 -0.14410043 ... -0.14400674 -0.14400674\n",
      "  -0.14400674]\n",
      " [-0.16299517 -0.16299517 -0.16299517 ... -0.12951872 -0.12951872\n",
      "  -0.12951872]\n",
      " [-0.1719513  -0.1719513  -0.1719513  ... -0.1433245  -0.1433245\n",
      "  -0.1433245 ]], shape=(65536, 40), dtype=float32)\n",
      "\n",
      "[INFO]: embedding_vector:\n",
      " ------------------------------tf.Tensor(\n",
      "[[-0.20234233 -0.20234233 -0.20234233 ... -0.20263647 -0.20263647\n",
      "  -0.20263647]\n",
      " [-0.21458414 -0.21458414 -0.21458414 ... -0.20683247 -0.20683247\n",
      "  -0.20683247]\n",
      " [-0.16928434 -0.16928434 -0.16928434 ... -0.22652037 -0.22652037\n",
      "  -0.22652037]\n",
      " ...\n",
      " [-0.19796263 -0.19796263 -0.19796263 ... -0.19235149 -0.19235149\n",
      "  -0.19235149]\n",
      " [-0.1892046  -0.1892046  -0.1892046  ... -0.23315167 -0.23315167\n",
      "  -0.23315167]\n",
      " [-0.16524336 -0.16524336 -0.16524336 ... -0.19004619 -0.19004619\n",
      "  -0.19004619]], shape=(65536, 40), dtype=float32) \n",
      "38 ------------------------------\n",
      "------------------------------ 42 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16781548 -0.16781548 -0.16781548 ... -0.17625847 -0.17625847\n",
      "  -0.17625847]\n",
      " [-0.1397596  -0.1397596  -0.1397596  ... -0.13837096 -0.13837096\n",
      "  -0.13837096]\n",
      " [-0.12085916 -0.12085916 -0.12085916 ... -0.15863958 -0.15863958\n",
      "  -0.15863958]\n",
      " ...\n",
      " [-0.17922549 -0.17922549 -0.17922549 ... -0.13088962 -0.13088962\n",
      "  -0.13088962]\n",
      " [-0.17063819 -0.17063819 -0.17063819 ... -0.13989024 -0.13989024\n",
      "  -0.13989024]\n",
      " [-0.17062347 -0.17062347 -0.17062347 ... -0.15872523 -0.15872523\n",
      "  -0.15872523]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22078812 -0.22078812 -0.22078812 ... -0.18950005 -0.18950005\n",
      "  -0.18950005]\n",
      " [-0.23366359 -0.23366359 -0.23366359 ... -0.22141077 -0.22141077\n",
      "  -0.22141077]\n",
      " [-0.21895395 -0.21895395 -0.21895395 ... -0.1915653  -0.1915653\n",
      "  -0.1915653 ]\n",
      " ...\n",
      " [-0.19114698 -0.19114698 -0.19114698 ... -0.21150537 -0.21150537\n",
      "  -0.21150537]\n",
      " [-0.21161428 -0.21161428 -0.21161428 ... -0.17058599 -0.17058599\n",
      "  -0.17058599]\n",
      " [-0.2138949  -0.2138949  -0.2138949  ... -0.18320766 -0.18320766\n",
      "  -0.18320766]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 39 ------------------------------\n",
      "------------------------------ 43 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.16485472 -0.16485472 -0.16485472 ... -0.2012884  -0.2012884\n",
      "  -0.2012884 ]\n",
      " [-0.16366625 -0.16366625 -0.16366625 ... -0.13152815 -0.13152815\n",
      "  -0.13152815]\n",
      " [-0.15201207 -0.15201207 -0.15201207 ... -0.14416565 -0.14416565\n",
      "  -0.14416565]\n",
      " ...\n",
      " [-0.2064325  -0.2064325  -0.2064325  ... -0.16895425 -0.16895425\n",
      "  -0.16895425]\n",
      " [-0.17918114 -0.17918114 -0.17918114 ... -0.18141751 -0.18141751\n",
      "  -0.18141751]\n",
      " [-0.15831333 -0.15831333 -0.15831333 ... -0.15296361 -0.15296361\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  -0.15296361]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.21763018 -0.21763018 -0.21763018 ... -0.21754324 -0.21754324\n",
      "  -0.21754324]\n",
      " [-0.19721556 -0.19721556 -0.19721556 ... -0.20997626 -0.20997626\n",
      "  -0.20997626]\n",
      " [-0.18108685 -0.18108685 -0.18108685 ... -0.17802556 -0.17802556\n",
      "  -0.17802556]\n",
      " ...\n",
      " [-0.17719774 -0.17719774 -0.17719774 ... -0.1929565  -0.1929565\n",
      "  -0.1929565 ]\n",
      " [-0.20807157 -0.20807157 -0.20807157 ... -0.18814465 -0.18814465\n",
      "  -0.18814465]\n",
      " [-0.21611315 -0.21611315 -0.21611315 ... -0.1911238  -0.1911238\n",
      "  -0.1911238 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 40 ------------------------------\n",
      "------------------------------ 44 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.1847917  -0.1847917  -0.1847917  ... -0.18993592 -0.18993592\n",
      "  -0.18993592]\n",
      " [-0.17649817 -0.17649817 -0.17649817 ... -0.17080042 -0.17080042\n",
      "  -0.17080042]\n",
      " [-0.18804726 -0.18804726 -0.18804726 ... -0.15337943 -0.15337943\n",
      "  -0.15337943]\n",
      " ...\n",
      " [-0.16575237 -0.16575237 -0.16575237 ... -0.16689557 -0.16689557\n",
      "  -0.16689557]\n",
      " [-0.16589029 -0.16589029 -0.16589029 ... -0.19078447 -0.19078447\n",
      "  -0.19078447]\n",
      " [-0.0958272  -0.0958272  -0.0958272  ... -0.15209657 -0.15209657\n",
      "  -0.15209657]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23361957 -0.23361957 -0.23361957 ... -0.22457814 -0.22457814\n",
      "  -0.22457814]\n",
      " [-0.26063687 -0.26063687 -0.26063687 ... -0.21892227 -0.21892227\n",
      "  -0.21892227]\n",
      " [-0.24077617 -0.24077617 -0.24077617 ... -0.24446452 -0.24446452\n",
      "  -0.24446452]\n",
      " ...\n",
      " [-0.2692767  -0.2692767  -0.2692767  ... -0.2200042  -0.2200042\n",
      "  -0.2200042 ]\n",
      " [-0.24632248 -0.24632248 -0.24632248 ... -0.22634555 -0.22634555\n",
      "  -0.22634555]\n",
      " [-0.22406167 -0.22406167 -0.22406167 ... -0.1985489  -0.1985489\n",
      "  -0.1985489 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 41 ------------------------------\n",
      "------------------------------ 45 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.20234235 -0.20234235 -0.20234235 ... -0.20263644 -0.20263644\n",
      "  -0.20263644]\n",
      " [-0.21458414 -0.21458414 -0.21458414 ... -0.20683241 -0.20683241\n",
      "  -0.20683241]\n",
      " [-0.1692844  -0.1692844  -0.1692844  ... -0.22652042 -0.22652042\n",
      "  -0.22652042]\n",
      " ...\n",
      " [-0.19796266 -0.19796266 -0.19796266 ... -0.19235143 -0.19235143\n",
      "  -0.19235143]\n",
      " [-0.18920456 -0.18920456 -0.18920456 ... -0.23315158 -0.23315158\n",
      "  -0.23315158]\n",
      " [-0.16524334 -0.16524334 -0.16524334 ... -0.19004619 -0.19004619\n",
      "  -0.19004619]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.25435817 -0.25435817 -0.25435817 ... -0.21289355 -0.21289355\n",
      "  -0.21289355]\n",
      " [-0.19415368 -0.19415368 -0.19415368 ... -0.25159675 -0.25159675\n",
      "  -0.25159675]\n",
      " [-0.21563295 -0.21563295 -0.21563295 ... -0.23470005 -0.23470005\n",
      "  -0.23470005]\n",
      " ...\n",
      " [-0.2295041  -0.2295041  -0.2295041  ... -0.2544933  -0.2544933\n",
      "  -0.2544933 ]\n",
      " [-0.23874322 -0.23874322 -0.23874322 ... -0.26085454 -0.26085454\n",
      "  -0.26085454]\n",
      " [-0.21130578 -0.21130578 -0.21130578 ... -0.23463786 -0.23463786\n",
      "  -0.23463786]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 42 ------------------------------\n",
      "------------------------------ 46 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22078814 -0.22078814 -0.22078814 ... -0.18950002 -0.18950002\n",
      "  -0.18950002]\n",
      " [-0.23366357 -0.23366357 -0.23366357 ... -0.22141083 -0.22141083\n",
      "  -0.22141083]\n",
      " [-0.21895395 -0.21895395 -0.21895395 ... -0.19156523 -0.19156523\n",
      "  -0.19156523]\n",
      " ...\n",
      " [-0.191147   -0.191147   -0.191147   ... -0.21150541 -0.21150541\n",
      "  -0.21150541]\n",
      " [-0.21161431 -0.21161431 -0.21161431 ... -0.170586   -0.170586\n",
      "  -0.170586  ]\n",
      " [-0.21389493 -0.21389493 -0.21389493 ... -0.18320759 -0.18320759\n",
      "  -0.18320759]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22544064 -0.22544064 -0.22544064 ... -0.23053777 -0.23053777\n",
      "  -0.23053777]\n",
      " [-0.26636156 -0.26636156 -0.26636156 ... -0.24495123 -0.24495123\n",
      "  -0.24495123]\n",
      " [-0.20872189 -0.20872189 -0.20872189 ... -0.25625733 -0.25625733\n",
      "  -0.25625733]\n",
      " ...\n",
      " [-0.23805548 -0.23805548 -0.23805548 ... -0.26428482 -0.26428482\n",
      "  -0.26428482]\n",
      " [-0.22804637 -0.22804637 -0.22804637 ... -0.2228002  -0.2228002\n",
      "  -0.2228002 ]\n",
      " [-0.2509238  -0.2509238  -0.2509238  ... -0.24291158 -0.24291158\n",
      "  -0.24291158]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 47 ------------------------------\n",
      "------------------------------ 43 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " [INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22355224 -0.22355224 -0.22355224 ... -0.2359108  -0.2359108\n",
      "  -0.2359108 ]\n",
      " [-0.27213806 -0.27213806 -0.27213806 ... -0.25591663 -0.25591663\n",
      "  -0.25591663]\n",
      " [-0.27956656 -0.27956656 -0.27956656 ... -0.24207915 -0.24207915\n",
      "  -0.24207915]\n",
      " ...\n",
      " [-0.24959457 -0.24959457 -0.24959457 ... -0.24118412 -0.24118412\n",
      "  -0.24118412]\n",
      " [-0.24258156 -0.24258156 -0.24258156 ... -0.30557442 -0.30557442\n",
      "  -0.30557442]\n",
      " [-0.23975593 -0.23975593 -0.23975593 ... -0.20874879 -0.20874879\n",
      "  -0.20874879]], shape=(65536, 40), dtype=float32)\n",
      "tf.Tensor(\n",
      "[[-0.21763018 -0.21763018 -0.21763018 ... -0.21754329 -0.21754329\n",
      "  -0.21754329]\n",
      " [-0.19721562 -0.19721562 -0.19721562 ... -0.2099763  -0.2099763\n",
      "  -0.2099763 ]\n",
      " [-0.1810869  -0.1810869  -0.1810869  ... -0.1780256  -0.1780256\n",
      "  -0.1780256 ]\n",
      " ...\n",
      " [-0.17719774 -0.17719774 -0.17719774 ... -0.19295655 -0.19295655\n",
      "  -0.19295655]\n",
      " [-0.20807157 -0.20807157 -0.20807157 ... -0.18814471 -0.18814471\n",
      "  -0.18814471]\n",
      " [-0.21611312 -0.21611312 -0.21611312 ... -0.19112383 -0.19112383\n",
      "  -0.19112383]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 48 ------------------------------\n",
      "------------------------------ 44 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.26985082 -0.26985082 -0.26985082 ... -0.24529496 -0.24529496\n",
      "  -0.24529496]\n",
      " [-0.25112605 -0.25112605 -0.25112605 ... -0.25174588 -0.25174588\n",
      "  -0.25174588]\n",
      " [-0.2261589  -0.2261589  -0.2261589  ... -0.23483409 -0.23483409\n",
      "  -0.23483409]\n",
      " ...\n",
      " [-0.21658464 -0.21658464 -0.21658464 ... -0.23378946 -0.23378946\n",
      "  -0.23378946]\n",
      " [-0.23269106 -0.23269106 -0.23269106 ... -0.1725443  -0.1725443\n",
      "  -0.1725443 ]\n",
      " [-0.25283694 -0.25283694 -0.25283694 ... -0.24298494 -0.24298494\n",
      "  -0.24298494]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23361962 -0.23361962 -0.23361962 ... -0.22457817 -0.22457817\n",
      "  -0.22457817]\n",
      " [-0.2606369  -0.2606369  -0.2606369  ... -0.21892226 -0.21892226\n",
      "  -0.21892226]\n",
      " [-0.24077623 -0.24077623 -0.24077623 ... -0.24446453 -0.24446453\n",
      "  -0.24446453]\n",
      " ...\n",
      " [-0.26927668 -0.26927668 -0.26927668 ... -0.2200041  -0.2200041\n",
      "  -0.2200041 ]\n",
      " [-0.24632253 -0.24632253 -0.24632253 ... -0.2263456  -0.2263456\n",
      "  -0.2263456 ]\n",
      " [-0.2240617  -0.2240617  -0.2240617  ... -0.19854891 -0.19854891\n",
      "  -0.19854891]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 49 ------------------------------\n",
      "------------------------------ 45 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23987098 -0.23987098 -0.23987098 ... -0.24169141 -0.24169141\n",
      "  -0.24169141]\n",
      " [-0.24237491 -0.24237491 -0.24237491 ... -0.23094192 -0.23094192\n",
      "  -0.23094192]\n",
      " [-0.23493645 -0.23493645 -0.23493645 ... -0.21154362 -0.21154362\n",
      "  -0.21154362]\n",
      " ...\n",
      " [-0.22897291 -0.22897291 -0.22897291 ... -0.25140733 -0.25140733\n",
      "  -0.25140733]\n",
      " [-0.25645626 -0.25645626 -0.25645626 ... -0.22097765 -0.22097765\n",
      "  -0.22097765]\n",
      " [-0.21199685 -0.21199685 -0.21199685 ... -0.2549315  -0.2549315\n",
      "  -0.2549315 ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.2543582  -0.2543582  -0.2543582  ... -0.21289355 -0.21289355\n",
      "  -0.21289355]\n",
      " [-0.19415367 -0.19415367 -0.19415367 ... -0.25159675 -0.25159675\n",
      "  -0.25159675]\n",
      " [-0.21563296 -0.21563296 -0.21563296 ... -0.2347001  -0.2347001\n",
      "  -0.2347001 ]\n",
      " ...\n",
      " [-0.22950411 -0.22950411 -0.22950411 ... -0.25449327 -0.25449327\n",
      "  -0.25449327]\n",
      " [-0.23874322 -0.23874322 -0.23874322 ... -0.26085454 -0.26085454\n",
      "  -0.26085454]\n",
      " [-0.21130578 -0.21130578 -0.21130578 ... -0.23463778 -0.23463778\n",
      "  -0.23463778]], shape=(65536, 40), dtype=float32)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------ 46 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22544068 -0.22544068 -0.22544068 ... -0.23053782 -0.23053782\n",
      "  -0.23053782]\n",
      " [-0.26636156 -0.26636156 -0.26636156 ... -0.24495119 -0.24495119\n",
      "  -0.24495119]\n",
      " [-0.20872188 -0.20872188 -0.20872188 ... -0.25625733 -0.25625733\n",
      "  -0.25625733]\n",
      " ...\n",
      " [-0.2380555  -0.2380555  -0.2380555  ... -0.26428485 -0.26428485\n",
      "  -0.26428485]\n",
      " [-0.2280464  -0.2280464  -0.2280464  ... -0.22280021 -0.22280021\n",
      "  -0.22280021]\n",
      " [-0.25092384 -0.25092384 -0.25092384 ... -0.2429116  -0.2429116\n",
      "  -0.2429116 ]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 47 ------------------------------\n",
      "[INFO] loadded from file ./sok_embedding_vectors_0.file\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.22355214 -0.22355214 -0.22355214 ... -0.23591082 -0.23591082\n",
      "  -0.23591082]\n",
      " [-0.27213818 -0.27213818 -0.27213818 ... -0.25591666 -0.25591666\n",
      "  -0.25591666]\n",
      " [-0.27956653 -0.27956653 -0.27956653 ... -0.24207921 -0.24207921\n",
      "  -0.24207921]\n",
      " ...\n",
      " [-0.24959451 -0.24959451 -0.24959451 ... -0.24118413 -0.24118413\n",
      "  -0.24118413]\n",
      " [-0.24258153 -0.24258153 -0.24258153 ... -0.30557433 -0.30557433\n",
      "  -0.30557433]\n",
      " [-0.23975603 -0.23975603 -0.23975603 ... -0.20874879 -0.20874879\n",
      "  -0.20874879]], shape=(65536, 40), dtype=float32)\n",
      "------------------------------ 48 ------------------------------\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.26985085 -0.26985085 -0.26985085 ... -0.24529503 -0.24529503\n",
      "  -0.24529503]\n",
      " [-0.25112605 -0.25112605 -0.25112605 ... -0.25174588 -0.25174588\n",
      "  -0.25174588]\n",
      " [-0.22615892 -0.22615892 -0.22615892 ... -0.23483418 -0.23483418\n",
      "  -0.23483418]\n",
      " ...\n",
      " [-0.21658465 -0.21658465 -0.21658465 ... -0.23378949 -0.23378949\n",
      "  -0.23378949]\n",
      " [-0.23269112 -0.23269112 -0.23269112 ... -0.17254438 -0.17254438\n",
      "  -0.17254438]\n",
      " [-0.252837   -0.252837   -0.252837   ... -0.242985   -0.242985\n",
      "  -0.242985  ]], shape=(65536, 40), dtype=float32)\n",
      "[INFO] loadded from file ./sok_embedding_vectors_1.file\n",
      "------------------------------ 49 ------------------------------\n",
      "\n",
      "[INFO]: With MultiWorkerMirroredStrategy, when 4 GPUs are used for each node and 8 GPUs in total, the embedding vectors obtained from TensorFlow and SOK are consistent for 50 iterations\n",
      "[INFO]: embedding_vector:\n",
      " tf.Tensor(\n",
      "[[-0.23987097 -0.23987097 -0.23987097 ... -0.24169141 -0.24169141\n",
      "  -0.24169141]\n",
      " [-0.24237491 -0.24237491 -0.24237491 ... -0.23094198 -0.23094198\n",
      "  -0.23094198]\n",
      " [-0.23493648 -0.23493648 -0.23493648 ... -0.21154368 -0.21154368\n",
      "  -0.21154368]\n",
      " ...\n",
      " [-0.22897288 -0.22897288 -0.22897288 ... -0.25140738 -0.25140738\n",
      "  -0.25140738]\n",
      " [-0.25645626 -0.25645626 -0.25645626 ... -0.22097768 -0.22097768\n",
      "  -0.22097768]\n",
      " [-0.21199685 -0.21199685 -0.21199685 ... -0.25493145 -0.25493145\n",
      "  -0.25493145]], shape=(65536, 40), dtype=float32)\n",
      "[INFO] loadded from file ./sok_embedding_vectors_0.file\n",
      "[INFO] loadded from file ./sok_embedding_vectors_1.file\n",
      "\n",
      "[INFO]: With MultiWorkerMirroredStrategy, when 4 GPUs are used for each node and 8 GPUs in total, the embedding vectors obtained from TensorFlow and SOK are consistent for 50 iterations\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Process\n",
    "\n",
    "processes = list()\n",
    "for task_id in range(args[\"worker_num\"]):\n",
    "    available_gpus = \",\".join([str(per_worker_gpu_num * task_id + i)\n",
    "                              for i in range(per_worker_gpu_num)])\n",
    "    print(\"[INFO]: on task: %d, its avaiable GPUs are: %s\" %(task_id, available_gpus))\n",
    "    \n",
    "    os.environ[\"CUDA_VISIBLE_DEVICES\"] = available_gpus\n",
    "    process = Process(target=compare_sok_with_tf, args=(args, task_id))\n",
    "    process.start()\n",
    "    processes.append(process)\n",
    "    \n",
    "    \n",
    "for process in processes:\n",
    "    if process.is_alive():\n",
    "        process.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7770eb44",
   "metadata": {},
   "source": [
    "If no exceptions and their embedding vectors are totally consistent, then a sentence similar to the following one will be printed.\n",
    "```shell\n",
    "\"[INFO]: With MultiWorkerMirroredStrategy, when args[\"local_gpu_num\"] GPUs are used for each node and total_gpu_num GPUs in total, the embedding vectors obtained from TensorFlow and SOK are consistent for 50 iterations\"\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
